2024-06-06T08:10:30.730 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-06T08:10:30.730 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --archive /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240 -- /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240/orig.config.yaml 2024-06-06T08:10:30.777 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-06T08:10:30.856 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240 branch: squid description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7743240' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 3d746ca5346f34eef7e826a1eb71e59e02b682ae owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5050 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 416/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 3d746ca5346f34eef7e826a1eb71e59e02b682ae targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4gQesikORNHjsDJYIfXduMpqN54GYdnI0YMJnNcI++aC0jbe9wixaObq6gZEBN+KH5oB6+h7HYtugcgF4BK5Y= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOhy76KJwnMhND554zLKCtrPR3KLvqFdH4SdkGZAuiClUDA9jhTwnpk2UWkG5WZnt+NqcW9e2iDOwPs3jtaoG14= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /home/teuthology/src/git.ceph.com_ceph_squid/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-05_21:24:16 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-06-06T08:10:30.856 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa; will attempt to use it 2024-06-06T08:10:30.857 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks 2024-06-06T08:10:30.857 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-06T08:10:31.021 INFO:teuthology.task.internal:Checking packages... 2024-06-06T08:10:31.042 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-06T08:10:31.042 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T08:10:31.043 INFO:teuthology.packaging:ref: None 2024-06-06T08:10:31.043 INFO:teuthology.packaging:tag: None 2024-06-06T08:10:31.043 INFO:teuthology.packaging:branch: squid 2024-06-06T08:10:31.043 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T08:10:31.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-06T08:10:31.420 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2960.g3d746ca5 2024-06-06T08:10:31.422 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-06T08:10:31.430 INFO:teuthology.task.internal:no buildpackages task found 2024-06-06T08:10:31.430 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-06T08:10:31.442 INFO:teuthology.task.internal:Saving configuration 2024-06-06T08:10:31.459 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-06T08:10:31.467 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-06T08:10:31.491 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-06 08:05:09.017011', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4gQesikORNHjsDJYIfXduMpqN54GYdnI0YMJnNcI++aC0jbe9wixaObq6gZEBN+KH5oB6+h7HYtugcgF4BK5Y='} 2024-06-06T08:10:31.509 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-06 08:05:09.018511', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOhy76KJwnMhND554zLKCtrPR3KLvqFdH4SdkGZAuiClUDA9jhTwnpk2UWkG5WZnt+NqcW9e2iDOwPs3jtaoG14='} 2024-06-06T08:10:31.509 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-06T08:10:31.517 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-06-06T08:10:31.517 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-06-06T08:10:31.517 INFO:teuthology.run_tasks:Running task console_log... 2024-06-06T08:10:31.578 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fba65d5fbe0>, signals=[15]) 2024-06-06T08:10:31.578 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-06T08:10:31.585 INFO:teuthology.task.internal:Opening connections... 2024-06-06T08:10:31.586 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-06-06T08:10:31.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:10:31.660 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-06-06T08:10:31.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:10:31.731 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-06T08:10:31.738 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-06-06T08:10:31.756 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-06-06T08:10:31.756 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-06-06T08:10:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-06T08:10:31.814 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-06-06T08:10:31.814 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-06T08:10:31.814 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-06T08:10:31.814 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-06T08:10:31.814 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-06-06T08:10:31.843 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-06-06T08:10:31.861 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-06-06T08:10:31.861 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-06-06T08:10:31.918 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-06-06T08:10:31.918 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-06-06T08:10:31.918 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-06-06T08:10:31.918 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-06T08:10:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-06T08:10:31.919 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-06-06T08:10:31.943 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-06T08:10:31.956 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-06T08:10:31.962 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-06T08:10:31.962 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-06-06T08:10:31.965 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-06-06T08:10:31.981 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-06T08:10:31.988 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-06T08:10:31.989 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-06-06T08:10:32.022 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-06-06T08:10:32.121 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-06T08:10:32.176 INFO:teuthology.run_tasks:Running task kernel... 2024-06-06T08:10:32.191 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-06T08:10:32.192 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-06T08:10:32.192 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-06T08:10:32.192 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-06T08:10:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T08:10:32.209 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-06-06T08:10:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T08:10:32.210 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-06-06T08:10:32.266 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-437.el9.x86_64 2024-06-06T08:10:32.266 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-437.el9.x86_64 2024-06-06T08:10:32.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-06-06T08:10:32.268 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-437.el9.x86_64 2024-06-06T08:10:32.268 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.14.0-437.el9.x86_64 2024-06-06T08:10:32.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2024-06-06T08:10:34.900 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.1 MB 00:01 2024-06-06T08:10:34.946 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.1 MB 00:01 2024-06-06T08:10:36.889 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-06-06T08:10:37.085 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-06-06T08:10:41.356 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.2 MB 00:00 2024-06-06T08:10:41.560 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.2 MB 00:00 2024-06-06T08:10:42.823 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 17 kB 00:00 2024-06-06T08:10:43.426 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 17 kB 00:00 2024-06-06T08:10:43.954 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-06-06T08:10:44.496 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-06-06T08:10:49.074 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-06-06T08:10:49.677 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-06-06T08:10:51.096 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-06T08:10:51.097 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-06T08:10:51.189 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:10:51.195 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:10:51.196 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-06-06T08:10:51.196 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:10:51.196 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:10:51.196 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-06T08:10:51.196 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-06T08:10:51.197 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-06T08:10:51.197 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-06T08:10:51.197 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-06T08:10:51.197 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:10:51.197 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:10:51.198 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:10:51.198 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-06-06T08:10:51.198 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:10:51.199 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 80 M 2024-06-06T08:10:51.199 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 124 M 2024-06-06T08:10:51.199 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:10:51.421 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:10:51.423 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:10:51.430 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:10:51.567 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 3.9 MB/s | 893 kB 00:00 2024-06-06T08:10:51.718 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-06T08:10:51.718 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-06T08:10:51.810 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:10:51.815 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-06-06T08:10:51.816 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:10:51.816 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 80 M 2024-06-06T08:10:51.816 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 124 M 2024-06-06T08:10:51.816 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:10:51.834 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-06-06T08:10:52.178 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:10:52.180 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:10:52.183 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:10:52.201 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 46 MB/s | 29 MB 00:00 2024-06-06T08:10:52.336 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 3.7 MB/s | 893 kB 00:00 2024-06-06T08:10:52.509 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-06-06T08:10:52.512 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:10:52.512 INFO:teuthology.orchestra.run.smithi086.stdout:Total 61 MB/s | 80 MB 00:01 2024-06-06T08:10:52.603 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-06-06T08:10:52.979 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 45 MB/s | 29 MB 00:00 2024-06-06T08:10:53.212 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:10:53.295 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-06-06T08:10:53.298 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:10:53.298 INFO:teuthology.orchestra.run.smithi154.stdout:Total 54 MB/s | 80 MB 00:01 2024-06-06T08:10:53.543 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:10:53.543 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:10:53.997 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:10:54.326 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:10:54.327 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:10:54.504 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:10:54.504 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:10:55.261 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:10:55.262 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:10:56.883 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:10:57.577 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-06T08:10:57.660 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T08:10:57.669 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:10:58.383 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-06T08:10:58.480 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T08:11:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T08:11:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T08:11:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T08:11:01.353 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T08:11:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T08:11:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:06.104 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T08:11:06.233 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:10.075 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:10.961 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:39.506 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:39.546 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:39.902 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:39.902 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-06T08:11:39.903 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T08:11:39.903 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T08:11:40.573 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:40.612 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:40.687 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:40.688 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:11:40.688 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:11:40.688 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-06T08:11:40.688 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-06T08:11:40.688 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-06T08:11:40.688 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-06T08:11:40.688 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:11:40.688 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:11:40.960 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:40.960 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-06T08:11:40.961 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T08:11:40.961 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T08:11:41.033 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T08:11:41.118 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T08:11:41.118 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T08:11:41.118 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-06T08:11:41.118 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-06T08:11:41.118 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-06T08:11:41.119 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-06-06T08:11:41.119 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-06-06T08:11:41.119 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T08:11:41.170 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T08:11:41.171 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T08:11:41.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-06-06T08:11:41.201 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-06-06T08:11:41.201 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-06-06T08:11:41.201 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-06-06T08:11:41.201 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-06-06T08:11:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-06-06T08:11:41.203 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:11:41.203 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-06T08:11:41.203 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-06T08:11:41.203 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-06T08:11:41.203 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-06-06T08:11:41.203 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:11:41.203 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-06-06T08:11:41.204 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-06T08:11:41.204 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-452.el9.x86_64 2024-06-06T08:11:41.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-06T08:11:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T08:11:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:11:41.814 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:11:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-06T08:11:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-06T08:11:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-06T08:11:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-06T08:11:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:11:41.814 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:11:41.842 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-06-06T08:11:42.184 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T08:11:42.251 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T08:11:42.251 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T08:11:42.251 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-06T08:11:42.251 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-06T08:11:42.251 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-06T08:11:42.252 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-06-06T08:11:42.252 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2024-06-06T08:11:42.252 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T08:11:42.304 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T08:11:42.304 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T08:11:42.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo rpm -qi grub2-tools 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Name : grub2-tools 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Epoch : 1 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Version : 2.06 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Release : 68.el9 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: x86_64 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Group : Unspecified 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Size : 8264254 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:License : GPLv3+ 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-06T08:11:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:Packager : builder@centos.org 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:Vendor : CentOS 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:Summary : Support tools for GRUB. 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:Description : 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:hardware devices. 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:11:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:This subpackage provides tools for support of all platforms. 2024-06-06T08:11:42.346 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-06T08:11:42.346 INFO:teuthology.task.kernel:Updating grub on smithi154 to boot 5.14.0-452.el9.x86_64 2024-06-06T08:11:42.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-06T08:11:42.959 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2024-06-06T08:11:43.276 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-06T08:11:43.305 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-06-06T08:11:43.308 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-06-06T08:11:43.322 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.z2EM2du0ms 2024-06-06T08:11:43.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.z2EM2du0ms 2024-06-06T08:11:43.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.z2EM2du0ms 2024-06-06T08:11:43.530 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.z2EM2du0ms is 6KB 2024-06-06T08:11:43.578 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.z2EM2du0ms 2024-06-06T08:11:43.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-06-06T08:11:43.653 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-06T08:11:43.653 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-06T08:11:43.654 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-06T08:11:43.654 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-06T08:11:43.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-06T08:11:43.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-06-06T08:11:44.370 INFO:teuthology.orchestra.run.smithi154.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-06T08:11:44.402 INFO:teuthology.orchestra.run.smithi154.stderr:done 2024-06-06T08:11:44.404 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-06-06T08:11:44.419 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.K6DZGszepd 2024-06-06T08:11:44.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.K6DZGszepd 2024-06-06T08:11:44.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.K6DZGszepd 2024-06-06T08:11:44.627 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.K6DZGszepd is 6KB 2024-06-06T08:11:44.675 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.K6DZGszepd 2024-06-06T08:11:44.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo /bin/ls /boot/loader/entries || true 2024-06-06T08:11:44.751 INFO:teuthology.orchestra.run.smithi154.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-06T08:11:44.751 INFO:teuthology.orchestra.run.smithi154.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-06T08:11:44.751 INFO:teuthology.orchestra.run.smithi154.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-06T08:11:44.752 INFO:teuthology.orchestra.run.smithi154.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-06T08:11:44.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-06T08:11:44.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2024-06-06T08:12:13.840 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-06-06T08:12:13.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-06T08:12:13.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:12:14.913 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-06-06T08:12:14.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-06-06T08:12:14.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:12:32.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-06-06T08:12:33.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-06-06T08:12:41.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-06T08:12:41.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:12:42.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-06-06T08:12:42.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:12:44.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-06-06T08:12:45.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-06-06T08:12:56.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-06T08:12:56.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:12:57.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-06-06T08:12:57.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:13:14.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-06-06T08:13:29.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-06T08:13:29.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:13:30.006 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-06-06T08:13:30.399 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-06T08:13:30.399 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-452.el9.x86_64"... 2024-06-06T08:13:30.399 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-06-06T08:13:30.414 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-452.el9.x86_64 2024-06-06T08:13:30.414 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-06T08:13:30.414 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-06T08:13:30.414 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-06T08:13:31.415 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-06-06T08:13:31.415 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-06T08:13:31.491 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-06-06T08:13:31.532 DEBUG:teuthology.parallel:result is None 2024-06-06T08:13:57.507 DEBUG:teuthology.orchestra.remote:timed out 2024-06-06T08:14:12.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-06-06T08:14:12.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:14:12.816 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-06-06T08:14:13.181 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-06-06T08:14:13.182 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-452.el9.x86_64"... 2024-06-06T08:14:13.182 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-06-06T08:14:13.198 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-452.el9.x86_64 2024-06-06T08:14:13.198 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-06T08:14:13.198 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-06T08:14:13.198 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-06T08:14:14.199 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-06-06T08:14:14.200 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-06T08:14:14.267 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2024-06-06T08:14:14.296 DEBUG:teuthology.parallel:result is None 2024-06-06T08:14:14.296 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-06T08:14:14.304 INFO:teuthology.task.internal:Creating test directory... 2024-06-06T08:14:14.304 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-06T08:14:14.308 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-06T08:14:14.327 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-06T08:14:14.335 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-06T08:14:14.360 INFO:teuthology.task.internal:Creating archive directory... 2024-06-06T08:14:14.360 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-06T08:14:14.368 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-06T08:14:14.487 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-06T08:14:14.495 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-06T08:14:14.495 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-06T08:14:14.498 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-06T08:14:14.534 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T08:14:14.534 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T08:14:14.545 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T08:14:14.546 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T08:14:14.548 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-06T08:14:14.555 INFO:teuthology.task.internal:Configuring sudo... 2024-06-06T08:14:14.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-06T08:14:14.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-06T08:14:14.616 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-06T08:14:14.625 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-06T08:14:14.625 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-06T08:14:14.658 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-06T08:14:14.674 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T08:14:14.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T08:14:14.808 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T08:14:14.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T08:14:14.931 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:14:14.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-06T08:14:14.997 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T08:14:15.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T08:14:15.084 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T08:14:15.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T08:14:15.209 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:14:15.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-06T08:14:15.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-06-06T08:14:15.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-06-06T08:14:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T08:14:15.358 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T08:14:15.706 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-06T08:14:15.714 INFO:teuthology.task.internal:Starting timer... 2024-06-06T08:14:15.714 INFO:teuthology.run_tasks:Running task pcp... 2024-06-06T08:14:15.726 INFO:teuthology.run_tasks:Running task selinux... 2024-06-06T08:14:15.736 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-06T08:14:15.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-06-06T08:14:15.799 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-06-06T08:14:15.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-06-06T08:14:15.847 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-06-06T08:14:15.849 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-06T08:14:15.849 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-06-06T08:14:15.879 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-06-06T08:14:15.880 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-06-06T08:14:15.923 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-06-06T08:14:15.924 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive'} 2024-06-06T08:14:15.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T08:14:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T08:14:15.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T08:14:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T08:14:15.989 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-06T08:14:15.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-06-06T08:14:16.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-06-06T08:14:16.056 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-06T08:14:16.067 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-06T08:14:16.070 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-06T08:14:16.195 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-06T08:14:16.206 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-06T08:14:16.210 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-06T08:18:33.636 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2024-06-06T08:18:33.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-06T08:18:33.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:18:33.713 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-06-06T08:18:33.797 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-06T08:18:33.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-06-06T08:18:33.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T08:18:33.874 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-06-06T08:18:33.962 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-06-06T08:18:33.963 INFO:teuthology.run_tasks:Running task clock... 2024-06-06T08:18:33.974 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-06T08:18:33.974 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-06T08:18:33.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T08:18:33.977 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-06T08:18:33.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T08:18:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-06T08:18:34.018 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-06T08:18:34.034 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-06T08:18:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-06-06T08:18:34.049 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-06T08:18:34.052 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-06-06T08:18:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-06T08:18:34.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-06-06T08:18:34.080 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-06T08:18:34.087 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-06-06T08:18:34.101 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-06T08:18:34.115 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-06T08:18:34.137 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-06-06T08:18:34.140 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T08:18:34.141 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-06-06T08:18:34.141 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:18:34.141 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:18:34.141 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:18:34.141 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:18:34.170 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-06-06T08:18:34.176 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T08:18:34.177 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-06-06T08:18:34.177 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:18:34.177 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:18:34.177 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:18:34.177 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:18:34.178 INFO:teuthology.run_tasks:Running task install... 2024-06-06T08:18:34.197 DEBUG:teuthology.task.install:project ceph 2024-06-06T08:18:34.198 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-06T08:18:34.198 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-06T08:18:34.198 INFO:teuthology.task.install:Using flavor: default 2024-06-06T08:18:34.203 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-06T08:18:34.203 INFO:teuthology.task.install:extra packages: [] 2024-06-06T08:18:34.204 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-06T08:18:34.204 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T08:18:34.204 INFO:teuthology.packaging:ref: None 2024-06-06T08:18:34.204 INFO:teuthology.packaging:tag: None 2024-06-06T08:18:34.204 INFO:teuthology.packaging:branch: quincy 2024-06-06T08:18:34.204 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T08:18:34.205 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-06T08:18:34.206 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-06T08:18:34.206 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T08:18:34.206 INFO:teuthology.packaging:ref: None 2024-06-06T08:18:34.206 INFO:teuthology.packaging:tag: None 2024-06-06T08:18:34.207 INFO:teuthology.packaging:branch: quincy 2024-06-06T08:18:34.207 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T08:18:34.207 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-06T08:18:34.415 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/ 2024-06-06T08:18:34.415 INFO:teuthology.task.install.rpm:Package version is 17.2.7-992.g6ee68927 2024-06-06T08:18:34.440 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/ 2024-06-06T08:18:34.440 INFO:teuthology.task.install.rpm:Package version is 17.2.7-992.g6ee68927 2024-06-06T08:18:34.556 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-06T08:18:34.556 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:18:34.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-06T08:18:34.584 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-06T08:18:34.584 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T08:18:34.584 INFO:teuthology.packaging:ref: None 2024-06-06T08:18:34.584 INFO:teuthology.packaging:tag: None 2024-06-06T08:18:34.584 INFO:teuthology.packaging:branch: quincy 2024-06-06T08:18:34.584 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T08:18:34.584 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-06T08:18:34.589 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-06T08:18:34.589 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:18:34.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-06T08:18:34.616 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-06T08:18:34.616 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T08:18:34.616 INFO:teuthology.packaging:ref: None 2024-06-06T08:18:34.616 INFO:teuthology.packaging:tag: None 2024-06-06T08:18:34.616 INFO:teuthology.packaging:branch: quincy 2024-06-06T08:18:34.616 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T08:18:34.616 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-06T08:18:34.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-06T08:18:34.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-06T08:18:34.725 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-06T08:18:34.756 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-06T08:18:34.791 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-06-06T08:18:34.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-06-06T08:18:34.820 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-06-06T08:18:34.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-06-06T08:18:35.029 INFO:teuthology.orchestra.run.smithi154.stdout:45 files removed 2024-06-06T08:18:35.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw 2024-06-06T08:18:35.057 INFO:teuthology.orchestra.run.smithi086.stdout:45 files removed 2024-06-06T08:18:35.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-06-06T08:18:35.760 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 185 kB/s | 77 kB 00:00 2024-06-06T08:18:35.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 191 kB/s | 77 kB 00:00 2024-06-06T08:18:36.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 39 kB/s | 11 kB 00:00 2024-06-06T08:18:36.144 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-06-06T08:18:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 5.5 kB/s | 1.8 kB 00:00 2024-06-06T08:18:36.469 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 6.3 kB/s | 1.8 kB 00:00 2024-06-06T08:18:37.334 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.1 MB 00:00 2024-06-06T08:18:37.346 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.1 MB 00:00 2024-06-06T08:18:39.086 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-06-06T08:18:39.131 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-06-06T08:18:43.340 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-06-06T08:18:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.2 MB 00:00 2024-06-06T08:18:44.820 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 17 kB 00:00 2024-06-06T08:18:44.999 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 17 kB 00:00 2024-06-06T08:18:46.131 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-06-06T08:18:46.310 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-06-06T08:18:51.321 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-06-06T08:18:51.616 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 8.5 kB/s | 1.7 kB 00:00 2024-06-06T08:18:53.737 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:18:53.750 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:18:53.750 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-06T08:18:53.750 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:17.2.7-992.g6ee68927.el9 ceph 14 M 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 3.1 M 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.9 M 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:17.2.7-992.g6ee68927.el9 ceph 5.0 M 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:17.2.7-992.g6ee68927.el9 ceph 22 M 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:17.2.7-992.g6ee68927.el9 ceph 25 k 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-06T08:18:53.751 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 637 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 458 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 4.9 M 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 45 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:17.2.7-992.g6ee68927.el9 ceph 117 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:17.2.7-992.g6ee68927.el9 ceph 156 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-06T08:18:53.752 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:17.2.7-992.g6ee68927.el9 ceph 321 k 2024-06-06T08:18:53.753 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 292 k 2024-06-06T08:18:53.753 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:17.2.7-992.g6ee68927.el9 ceph 99 k 2024-06-06T08:18:53.753 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-06T08:18:53.753 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:18:53.753 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:18:53.753 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:18:53.753 INFO:teuthology.orchestra.run.smithi086.stdout:Install 22 Packages 2024-06-06T08:18:53.753 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-06-06T08:18:53.753 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:18:53.754 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 58 M 2024-06-06T08:18:53.754 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:18:54.015 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:18:54.018 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:18:54.018 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-06-06T08:18:54.018 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:18:54.018 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:18:54.018 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:17.2.7-992.g6ee68927.el9 ceph 14 M 2024-06-06T08:18:54.018 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 3.1 M 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.9 M 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:17.2.7-992.g6ee68927.el9 ceph 5.0 M 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:17.2.7-992.g6ee68927.el9 ceph 22 M 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:17.2.7-992.g6ee68927.el9 ceph 25 k 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 637 k 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 458 k 2024-06-06T08:18:54.019 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 4.9 M 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 45 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:17.2.7-992.g6ee68927.el9 ceph 117 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:17.2.7-992.g6ee68927.el9 ceph 156 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:17.2.7-992.g6ee68927.el9 ceph 321 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 292 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:17.2.7-992.g6ee68927.el9 ceph 99 k 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:18:54.020 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:18:54.021 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:18:54.021 INFO:teuthology.orchestra.run.smithi154.stdout:Install 22 Packages 2024-06-06T08:18:54.021 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 2 Packages 2024-06-06T08:18:54.021 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:18:54.021 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 58 M 2024-06-06T08:18:54.021 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:18:55.214 INFO:teuthology.orchestra.run.smithi086.stdout:(1/24): ceph-radosgw-17.2.7-992.g6ee68927.el9.x 11 MB/s | 14 MB 00:01 2024-06-06T08:18:55.280 INFO:teuthology.orchestra.run.smithi086.stdout:(2/24): ceph-selinux-17.2.7-992.g6ee68927.el9.x 377 kB/s | 25 kB 00:00 2024-06-06T08:18:55.414 INFO:teuthology.orchestra.run.smithi086.stdout:(3/24): ceph-base-17.2.7-992.g6ee68927.el9.x86_ 3.3 MB/s | 5.0 MB 00:01 2024-06-06T08:18:55.456 INFO:teuthology.orchestra.run.smithi086.stdout:(4/24): libcephfs2-17.2.7-992.g6ee68927.el9.x86 3.6 MB/s | 637 kB 00:00 2024-06-06T08:18:55.539 INFO:teuthology.orchestra.run.smithi086.stdout:(5/24): libradosstriper1-17.2.7-992.g6ee68927.e 3.6 MB/s | 458 kB 00:00 2024-06-06T08:18:55.706 INFO:teuthology.orchestra.run.smithi086.stdout:(6/24): python3-ceph-argparse-17.2.7-992.g6ee68 271 kB/s | 45 kB 00:00 2024-06-06T08:18:55.832 INFO:teuthology.orchestra.run.smithi086.stdout:(7/24): python3-ceph-common-17.2.7-992.g6ee6892 939 kB/s | 117 kB 00:00 2024-06-06T08:18:55.899 INFO:teuthology.orchestra.run.smithi086.stdout:(8/24): librgw2-17.2.7-992.g6ee68927.el9.x86_64 11 MB/s | 4.9 MB 00:00 2024-06-06T08:18:55.933 INFO:teuthology.orchestra.run.smithi086.stdout:(9/24): python3-cephfs-17.2.7-992.g6ee68927.el9 1.5 MB/s | 156 kB 00:00 2024-06-06T08:18:55.983 INFO:teuthology.orchestra.run.smithi086.stdout:(10/24): python3-rados-17.2.7-992.g6ee68927.el9 3.7 MB/s | 321 kB 00:00 2024-06-06T08:18:56.125 INFO:teuthology.orchestra.run.smithi086.stdout:(11/24): python3-rgw-17.2.7-992.g6ee68927.el9.x 700 kB/s | 99 kB 00:00 2024-06-06T08:18:56.150 INFO:teuthology.orchestra.run.smithi086.stdout:(12/24): python3-rbd-17.2.7-992.g6ee68927.el9.x 1.3 MB/s | 292 kB 00:00 2024-06-06T08:18:56.206 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:18:56.211 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:18:56.358 INFO:teuthology.orchestra.run.smithi086.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 217 kB/s | 45 kB 00:00 2024-06-06T08:18:56.362 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:18:56.442 INFO:teuthology.orchestra.run.smithi086.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 505 kB/s | 160 kB 00:00 2024-06-06T08:18:56.568 INFO:teuthology.orchestra.run.smithi086.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-06-06T08:18:56.618 INFO:teuthology.orchestra.run.smithi086.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-06-06T08:18:56.643 INFO:teuthology.orchestra.run.smithi086.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-06-06T08:18:56.668 INFO:teuthology.orchestra.run.smithi086.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-06T08:18:56.693 INFO:teuthology.orchestra.run.smithi086.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.0 MB/s | 309 kB 00:00 2024-06-06T08:18:56.711 INFO:teuthology.orchestra.run.smithi086.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-06-06T08:18:56.733 INFO:teuthology.orchestra.run.smithi154.stdout:(1/24): ceph-common-17.2.7-992.g6ee68927.el9.x8 9.4 MB/s | 22 MB 00:02 2024-06-06T08:18:56.809 INFO:teuthology.orchestra.run.smithi154.stdout:(2/24): ceph-base-17.2.7-992.g6ee68927.el9.x86_ 2.0 MB/s | 5.0 MB 00:02 2024-06-06T08:18:56.825 INFO:teuthology.orchestra.run.smithi154.stdout:(3/24): ceph-selinux-17.2.7-992.g6ee68927.el9.x 274 kB/s | 25 kB 00:00 2024-06-06T08:18:56.844 INFO:teuthology.orchestra.run.smithi086.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-06-06T08:18:56.926 INFO:teuthology.orchestra.run.smithi154.stdout:(4/24): libradosstriper1-17.2.7-992.g6ee68927.e 4.5 MB/s | 458 kB 00:00 2024-06-06T08:18:56.993 INFO:teuthology.orchestra.run.smithi154.stdout:(5/24): libcephfs2-17.2.7-992.g6ee68927.el9.x86 3.4 MB/s | 637 kB 00:00 2024-06-06T08:18:57.093 INFO:teuthology.orchestra.run.smithi154.stdout:(6/24): python3-ceph-argparse-17.2.7-992.g6ee68 451 kB/s | 45 kB 00:00 2024-06-06T08:18:57.210 INFO:teuthology.orchestra.run.smithi154.stdout:(7/24): python3-ceph-common-17.2.7-992.g6ee6892 1.0 MB/s | 117 kB 00:00 2024-06-06T08:18:57.231 INFO:teuthology.orchestra.run.smithi086.stdout:(22/24): librbd1-17.2.7-992.g6ee68927.el9.x86_6 7.7 MB/s | 2.9 MB 00:00 2024-06-06T08:18:57.270 INFO:teuthology.orchestra.run.smithi086.stdout:(23/24): librados2-17.2.7-992.g6ee68927.el9.x86 5.5 MB/s | 3.1 MB 00:00 2024-06-06T08:18:57.285 INFO:teuthology.orchestra.run.smithi154.stdout:(8/24): python3-cephfs-17.2.7-992.g6ee68927.el9 2.0 MB/s | 156 kB 00:00 2024-06-06T08:18:57.436 INFO:teuthology.orchestra.run.smithi154.stdout:(9/24): librgw2-17.2.7-992.g6ee68927.el9.x86_64 9.6 MB/s | 4.9 MB 00:00 2024-06-06T08:18:57.461 INFO:teuthology.orchestra.run.smithi154.stdout:(10/24): python3-rados-17.2.7-992.g6ee68927.el9 1.8 MB/s | 321 kB 00:00 2024-06-06T08:18:57.486 INFO:teuthology.orchestra.run.smithi154.stdout:(11/24): python3-rbd-17.2.7-992.g6ee68927.el9.x 5.7 MB/s | 292 kB 00:00 2024-06-06T08:18:57.537 INFO:teuthology.orchestra.run.smithi154.stdout:(12/24): python3-rgw-17.2.7-992.g6ee68927.el9.x 1.3 MB/s | 99 kB 00:00 2024-06-06T08:18:57.540 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:18:57.577 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:18:57.645 INFO:teuthology.orchestra.run.smithi154.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 428 kB/s | 45 kB 00:00 2024-06-06T08:18:57.671 INFO:teuthology.orchestra.run.smithi154.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 872 kB/s | 160 kB 00:00 2024-06-06T08:18:57.671 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:18:57.672 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] lttng-ust-2.12.0-6.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lttng-ust-2.12.0-6.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:18:57.738 INFO:teuthology.orchestra.run.smithi154.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-06-06T08:18:57.763 INFO:teuthology.orchestra.run.smithi154.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-06-06T08:18:57.780 INFO:teuthology.orchestra.run.smithi154.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-06-06T08:18:57.805 INFO:teuthology.orchestra.run.smithi154.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-06T08:18:57.831 INFO:teuthology.orchestra.run.smithi154.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.5 MB/s | 309 kB 00:00 2024-06-06T08:18:57.856 INFO:teuthology.orchestra.run.smithi154.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-06T08:18:57.923 INFO:teuthology.orchestra.run.smithi154.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-06-06T08:18:58.181 INFO:teuthology.orchestra.run.smithi154.stdout:(22/24): librbd1-17.2.7-992.g6ee68927.el9.x86_6 11 MB/s | 2.9 MB 00:00 2024-06-06T08:18:58.415 INFO:teuthology.orchestra.run.smithi154.stdout:(23/24): librados2-17.2.7-992.g6ee68927.el9.x86 5.5 MB/s | 3.1 MB 00:00 2024-06-06T08:18:59.520 INFO:teuthology.orchestra.run.smithi086.stdout:(24/24): ceph-common-17.2.7-992.g6ee68927.el9.x 4.0 MB/s | 22 MB 00:05 2024-06-06T08:18:59.525 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:18:59.526 INFO:teuthology.orchestra.run.smithi086.stdout:Total 10 MB/s | 58 MB 00:05 2024-06-06T08:18:59.590 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:18:59.623 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:18:59.623 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:19:00.164 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:19:00.164 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:19:00.340 INFO:teuthology.orchestra.run.smithi154.stdout:(24/24): ceph-radosgw-17.2.7-992.g6ee68927.el9. 2.3 MB/s | 14 MB 00:06 2024-06-06T08:19:00.345 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:00.346 INFO:teuthology.orchestra.run.smithi154.stdout:Total 9.1 MB/s | 58 MB 00:06 2024-06-06T08:19:00.409 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:19:00.443 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:19:00.443 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:19:00.976 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:19:00.976 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:19:01.352 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:19:01.733 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-06T08:19:01.780 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T08:19:01.880 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T08:19:01.959 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 3/26 2024-06-06T08:19:01.990 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T08:19:02.105 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T08:19:02.141 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:19:02.153 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-06T08:19:02.195 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-06T08:19:02.534 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-06T08:19:02.626 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T08:19:02.630 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-06T08:19:02.677 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 8/26 2024-06-06T08:19:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T08:19:02.731 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 8/26 2024-06-06T08:19:02.759 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-06T08:19:02.802 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x 10/26 2024-06-06T08:19:02.813 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 3/26 2024-06-06T08:19:02.839 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 11/26 2024-06-06T08:19:02.886 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T08:19:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 12/26 2024-06-06T08:19:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T08:19:02.977 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T08:19:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T08:19:03.019 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-06T08:19:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-06T08:19:03.049 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-06T08:19:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-06T08:19:03.103 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-06T08:19:03.399 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-06T08:19:03.442 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 18/26 2024-06-06T08:19:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-06T08:19:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 8/26 2024-06-06T08:19:03.530 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 18/26 2024-06-06T08:19:03.560 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 8/26 2024-06-06T08:19:03.596 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-06T08:19:03.605 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 19/26 2024-06-06T08:19:03.636 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86 20/26 2024-06-06T08:19:03.639 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x 10/26 2024-06-06T08:19:03.681 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 11/26 2024-06-06T08:19:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 12/26 2024-06-06T08:19:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T08:19:03.832 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T08:19:03.872 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-06T08:19:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-06T08:19:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-06T08:19:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-06T08:19:04.279 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 18/26 2024-06-06T08:19:04.352 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 18/26 2024-06-06T08:19:04.425 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 19/26 2024-06-06T08:19:04.457 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86 20/26 2024-06-06T08:19:06.015 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T08:19:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T08:19:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T08:19:06.628 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 22/26 2024-06-06T08:19:06.715 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 22/26 2024-06-06T08:19:06.716 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-06T08:19:06.716 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-06T08:19:06.716 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:06.746 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T08:19:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T08:19:06.912 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T08:19:07.445 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T08:19:07.483 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 22/26 2024-06-06T08:19:07.581 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 22/26 2024-06-06T08:19:07.581 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-06T08:19:07.581 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-06T08:19:07.581 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:07.609 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T08:19:16.114 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T08:19:16.114 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-06-06T08:19:16.114 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-06-06T08:19:16.114 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-06-06T08:19:16.114 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-06-06T08:19:16.114 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-06-06T08:19:16.115 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-06-06T08:19:16.115 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-06-06T08:19:16.115 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:17.053 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T08:19:17.053 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-06-06T08:19:17.053 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-06-06T08:19:17.053 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-06-06T08:19:17.053 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-06-06T08:19:17.054 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-06-06T08:19:17.054 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2024-06-06T08:19:17.054 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-06-06T08:19:17.054 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 24/26 2024-06-06T08:19:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 24/26 2024-06-06T08:19:17.477 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:17.477 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-06T08:19:17.477 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T08:19:17.478 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T08:19:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-06T08:19:17.568 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-06T08:19:17.568 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T08:19:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 24/26 2024-06-06T08:19:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 24/26 2024-06-06T08:19:18.388 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:18.388 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-06T08:19:18.388 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T08:19:18.388 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T08:19:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-06T08:19:18.481 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-06T08:19:18.481 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T08:19:18.693 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T08:19:18.694 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 1/26 2024-06-06T08:19:18.694 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T08:19:18.694 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 3/26 2024-06-06T08:19:18.694 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T08:19:18.694 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 5/26 2024-06-06T08:19:18.694 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 6/26 2024-06-06T08:19:18.694 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 7/26 2024-06-06T08:19:18.694 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x 8/26 2024-06-06T08:19:18.694 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86 9/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 10/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 11/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 12/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-06T08:19:18.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-06T08:19:18.696 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-06T08:19:18.696 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-06T08:19:18.696 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T08:19:18.696 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-06T08:19:18.696 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 25/26 2024-06-06T08:19:19.305 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T08:19:19.305 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:19.305 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-06-06T08:19:19.305 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.306 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:19.307 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:19:19.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-06-06T08:19:19.826 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T08:19:19.826 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 1/26 2024-06-06T08:19:19.826 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T08:19:19.826 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 3/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 5/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 6/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 7/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x 8/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86 9/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 10/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 11/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 12/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-06T08:19:19.827 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-06T08:19:19.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-06T08:19:19.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-06T08:19:19.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-06T08:19:19.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-06T08:19:19.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-06T08:19:19.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-06T08:19:19.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T08:19:19.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-06T08:19:19.828 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 25/26 2024-06-06T08:19:20.231 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:29 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:20.350 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:19:20.350 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:17.2.7-992.g6ee68927.el9 ceph 48 M 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:17.2.7-992.g6ee68927.el9 ceph 163 k 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:19:20.351 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:20.352 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-06-06T08:19:20.352 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:20.352 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 48 M 2024-06-06T08:19:20.352 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 192 M 2024-06-06T08:19:20.352 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:19:20.465 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] socat-1.7.4.1-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:20.572 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-06-06T08:19:20.575 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:20.614 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-06T08:19:20.639 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): libcephsqlite-17.2.7-992.g6ee68927.el9.x 652 kB/s | 163 kB 00:00 2024-06-06T08:19:20.660 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T08:19:20.660 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-06T08:19:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-06T08:19:20.662 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:20.663 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:19:21.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-test 2024-06-06T08:19:21.591 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:30 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:21.714 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:19:21.714 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:17.2.7-992.g6ee68927.el9 ceph 48 M 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:17.2.7-992.g6ee68927.el9 ceph 163 k 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:21.715 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-06-06T08:19:21.716 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:21.716 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 48 M 2024-06-06T08:19:21.716 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 192 M 2024-06-06T08:19:21.717 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:19:21.834 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] socat-1.7.4.1-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:22.006 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 305 kB 00:00 2024-06-06T08:19:22.030 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): libcephsqlite-17.2.7-992.g6ee68927.el9.x 594 kB/s | 163 kB 00:00 2024-06-06T08:19:22.030 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:22.081 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 845 kB/s | 64 kB 00:00 2024-06-06T08:19:25.447 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): ceph-test-17.2.7-992.g6ee68927.el9.x86_6 9.5 MB/s | 48 MB 00:05 2024-06-06T08:19:25.450 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:25.451 INFO:teuthology.orchestra.run.smithi086.stdout:Total 9.5 MB/s | 48 MB 00:05 2024-06-06T08:19:25.478 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:19:25.498 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:19:25.498 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:19:25.861 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:19:25.862 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:19:26.130 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): ceph-test-17.2.7-992.g6ee68927.el9.x86_6 11 MB/s | 48 MB 00:04 2024-06-06T08:19:26.133 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:26.134 INFO:teuthology.orchestra.run.smithi154.stdout:Total 11 MB/s | 48 MB 00:04 2024-06-06T08:19:26.161 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:19:26.181 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:19:26.182 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:19:26.390 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:19:26.465 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-06T08:19:26.505 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-06T08:19:26.532 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 3/4 2024-06-06T08:19:26.546 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:19:26.547 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:19:27.069 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:19:27.141 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-06T08:19:27.188 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-06T08:19:27.216 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 3/4 2024-06-06T08:19:30.917 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 3/4 2024-06-06T08:19:30.997 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 4/4 2024-06-06T08:19:31.680 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 3/4 2024-06-06T08:19:31.764 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 4/4 2024-06-06T08:19:33.153 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 4/4 2024-06-06T08:19:33.153 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 1/4 2024-06-06T08:19:33.153 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 2/4 2024-06-06T08:19:33.153 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-06T08:19:33.450 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-06T08:19:33.450 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:33.450 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:19:33.450 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:33.450 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:33.450 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-06T08:19:33.450 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-06T08:19:33.451 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:33.451 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:19:33.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-06-06T08:19:34.031 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 4/4 2024-06-06T08:19:34.031 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 1/4 2024-06-06T08:19:34.031 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 2/4 2024-06-06T08:19:34.031 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-06T08:19:34.286 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:43 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-06T08:19:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:34.307 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:19:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-06T08:19:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-06T08:19:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:34.307 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:19:34.416 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:17.2.7-992.g6ee68927.el9 ceph 6.5 k 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.1 M 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:17.2.7-992.g6ee68927.el9 ceph 1.4 M 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 240 k 2024-06-06T08:19:34.419 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:17.2.7-992.g6ee68927.el9 ceph 3.9 M 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 15 M 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-06T08:19:34.420 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-06T08:19:34.421 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout:Install 33 Packages 2024-06-06T08:19:34.422 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:34.423 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 27 M 2024-06-06T08:19:34.423 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 96 M 2024-06-06T08:19:34.423 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:19:34.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph 2024-06-06T08:19:35.190 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:44 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:35.248 INFO:teuthology.orchestra.run.smithi086.stdout:(1/33): ceph-17.2.7-992.g6ee68927.el9.x86_64.rp 30 kB/s | 6.5 kB 00:00 2024-06-06T08:19:35.321 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:19:35.323 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:35.323 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:17.2.7-992.g6ee68927.el9 ceph 6.5 k 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.1 M 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:17.2.7-992.g6ee68927.el9 ceph 1.4 M 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 240 k 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:17.2.7-992.g6ee68927.el9 ceph 3.9 M 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 15 M 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-06T08:19:35.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-06T08:19:35.325 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-06T08:19:35.326 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:35.327 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:19:35.327 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:35.327 INFO:teuthology.orchestra.run.smithi154.stdout:Install 33 Packages 2024-06-06T08:19:35.327 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:35.327 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 27 M 2024-06-06T08:19:35.327 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 96 M 2024-06-06T08:19:35.328 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:19:35.565 INFO:teuthology.orchestra.run.smithi086.stdout:(2/33): ceph-mds-17.2.7-992.g6ee68927.el9.x86_6 3.9 MB/s | 2.1 MB 00:00 2024-06-06T08:19:35.632 INFO:teuthology.orchestra.run.smithi086.stdout:(3/33): ceph-mgr-17.2.7-992.g6ee68927.el9.x86_6 2.3 MB/s | 1.4 MB 00:00 2024-06-06T08:19:36.132 INFO:teuthology.orchestra.run.smithi086.stdout:(4/33): ceph-mgr-modules-core-17.2.7-992.g6ee68 480 kB/s | 240 kB 00:00 2024-06-06T08:19:36.168 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:36.191 INFO:teuthology.orchestra.run.smithi154.stdout:(1/33): ceph-17.2.7-992.g6ee68927.el9.x86_64.rp 23 kB/s | 6.5 kB 00:00 2024-06-06T08:19:36.274 INFO:teuthology.orchestra.run.smithi086.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-06-06T08:19:36.407 INFO:teuthology.orchestra.run.smithi154.stdout:(2/33): ceph-mgr-17.2.7-992.g6ee68927.el9.x86_6 2.8 MB/s | 1.4 MB 00:00 2024-06-06T08:19:36.424 INFO:teuthology.orchestra.run.smithi086.stdout:(6/33): ceph-osd-17.2.7-992.g6ee68927.el9.x86_6 18 MB/s | 15 MB 00:00 2024-06-06T08:19:36.425 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:36.425 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:19:36.491 INFO:teuthology.orchestra.run.smithi086.stdout:(7/33): python3-cryptography-36.0.1-4.el9.x86_6 5.7 MB/s | 1.2 MB 00:00 2024-06-06T08:19:36.517 INFO:teuthology.orchestra.run.smithi086.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-06-06T08:19:36.542 INFO:teuthology.orchestra.run.smithi086.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-06-06T08:19:36.584 INFO:teuthology.orchestra.run.smithi086.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-06-06T08:19:36.609 INFO:teuthology.orchestra.run.smithi086.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-06-06T08:19:36.609 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:19:36.612 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:36.684 INFO:teuthology.orchestra.run.smithi086.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 463 kB/s | 35 kB 00:00 2024-06-06T08:19:36.687 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-pytz-2021.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pytz-2021.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:19:36.750 INFO:teuthology.orchestra.run.smithi154.stdout:(3/33): ceph-mon-17.2.7-992.g6ee68927.el9.x86_6 7.0 MB/s | 3.9 MB 00:00 2024-06-06T08:19:36.751 INFO:teuthology.orchestra.run.smithi086.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 763 kB/s | 51 kB 00:00 2024-06-06T08:19:36.776 INFO:teuthology.orchestra.run.smithi086.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 895 kB/s | 172 kB 00:00 2024-06-06T08:19:36.793 INFO:teuthology.orchestra.run.smithi086.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 997 kB/s | 42 kB 00:00 2024-06-06T08:19:36.808 INFO:teuthology.orchestra.run.smithi154.stdout:(4/33): ceph-mds-17.2.7-992.g6ee68927.el9.x86_6 2.3 MB/s | 2.1 MB 00:00 2024-06-06T08:19:36.810 INFO:teuthology.orchestra.run.smithi086.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-06-06T08:19:36.833 INFO:teuthology.orchestra.run.smithi154.stdout:(5/33): ceph-mgr-modules-core-17.2.7-992.g6ee68 2.8 MB/s | 240 kB 00:00 2024-06-06T08:19:36.869 INFO:teuthology.orchestra.run.smithi086.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-06-06T08:19:36.885 INFO:teuthology.orchestra.run.smithi086.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 753 kB/s | 11 kB 00:00 2024-06-06T08:19:36.887 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:36.893 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:36.903 INFO:teuthology.orchestra.run.smithi086.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-06-06T08:19:36.928 INFO:teuthology.orchestra.run.smithi086.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-06-06T08:19:36.954 INFO:teuthology.orchestra.run.smithi086.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-06-06T08:19:37.003 INFO:teuthology.orchestra.run.smithi086.stdout:(22/33): ceph-mon-17.2.7-992.g6ee68927.el9.x86_ 2.2 MB/s | 3.9 MB 00:01 2024-06-06T08:19:37.029 INFO:teuthology.orchestra.run.smithi086.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-06-06T08:19:37.033 INFO:teuthology.orchestra.run.smithi154.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-06-06T08:19:37.034 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:19:37.054 INFO:teuthology.orchestra.run.smithi086.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-06-06T08:19:37.076 INFO:teuthology.orchestra.run.smithi154.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-06-06T08:19:37.104 INFO:teuthology.orchestra.run.smithi086.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-06-06T08:19:37.117 INFO:teuthology.orchestra.run.smithi154.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 4.7 MB/s | 1.2 MB 00:00 2024-06-06T08:19:37.129 INFO:teuthology.orchestra.run.smithi086.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 723 kB/s | 16 kB 00:00 2024-06-06T08:19:37.147 INFO:teuthology.orchestra.run.smithi154.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-06-06T08:19:37.154 INFO:teuthology.orchestra.run.smithi086.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-06-06T08:19:37.168 INFO:teuthology.orchestra.run.smithi154.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-06-06T08:19:37.180 INFO:teuthology.orchestra.run.smithi086.stdout:(28/33): python3-more-itertools-8.12.0-2.el9.no 521 kB/s | 79 kB 00:00 2024-06-06T08:19:37.193 INFO:teuthology.orchestra.run.smithi154.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-06-06T08:19:37.194 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:19:37.197 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:37.214 INFO:teuthology.orchestra.run.smithi086.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 606 kB/s | 36 kB 00:00 2024-06-06T08:19:37.239 INFO:teuthology.orchestra.run.smithi086.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 821 kB/s | 90 kB 00:00 2024-06-06T08:19:37.264 INFO:teuthology.orchestra.run.smithi086.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.3 MB/s | 427 kB 00:00 2024-06-06T08:19:37.290 INFO:teuthology.orchestra.run.smithi086.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-06-06T08:19:37.315 INFO:teuthology.orchestra.run.smithi086.stdout:(33/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-06-06T08:19:37.318 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:37.319 INFO:teuthology.orchestra.run.smithi086.stdout:Total 9.4 MB/s | 27 MB 00:02 2024-06-06T08:19:37.352 INFO:teuthology.orchestra.run.smithi154.stdout:(12/33): ceph-osd-17.2.7-992.g6ee68927.el9.x86_ 16 MB/s | 15 MB 00:00 2024-06-06T08:19:37.377 INFO:teuthology.orchestra.run.smithi154.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 189 kB/s | 35 kB 00:00 2024-06-06T08:19:37.378 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-pytz-2021.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pytz-2021.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:19:37.378 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-toml-0.10.2-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-toml-0.10.2-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:19:37.411 INFO:teuthology.orchestra.run.smithi154.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 710 kB/s | 172 kB 00:00 2024-06-06T08:19:37.432 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:19:37.436 INFO:teuthology.orchestra.run.smithi154.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 608 kB/s | 51 kB 00:00 2024-06-06T08:19:37.461 INFO:teuthology.orchestra.run.smithi154.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 863 kB/s | 43 kB 00:00 2024-06-06T08:19:37.468 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:19:37.468 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:19:37.487 INFO:teuthology.orchestra.run.smithi154.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-06-06T08:19:37.512 INFO:teuthology.orchestra.run.smithi154.stdout:(18/33): python3-toml-0.10.2-6.el9.noarch.rpm 309 kB/s | 42 kB 00:00 2024-06-06T08:19:37.537 INFO:teuthology.orchestra.run.smithi154.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-06-06T08:19:37.588 INFO:teuthology.orchestra.run.smithi154.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-06-06T08:19:37.621 INFO:teuthology.orchestra.run.smithi154.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 398 kB/s | 23 kB 00:00 2024-06-06T08:19:37.646 INFO:teuthology.orchestra.run.smithi154.stdout:(22/33): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-06-06T08:19:37.663 INFO:teuthology.orchestra.run.smithi154.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 265 kB/s | 19 kB 00:00 2024-06-06T08:19:37.689 INFO:teuthology.orchestra.run.smithi154.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-06-06T08:19:37.714 INFO:teuthology.orchestra.run.smithi154.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 916 kB/s | 46 kB 00:00 2024-06-06T08:19:37.739 INFO:teuthology.orchestra.run.smithi154.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-06-06T08:19:37.756 INFO:teuthology.orchestra.run.smithi154.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-06-06T08:19:37.781 INFO:teuthology.orchestra.run.smithi154.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-06-06T08:19:37.798 INFO:teuthology.orchestra.run.smithi154.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 858 kB/s | 36 kB 00:00 2024-06-06T08:19:37.824 INFO:teuthology.orchestra.run.smithi154.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-06-06T08:19:37.828 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:19:37.828 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:19:37.849 INFO:teuthology.orchestra.run.smithi154.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-06-06T08:19:37.874 INFO:teuthology.orchestra.run.smithi154.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-06-06T08:19:37.908 INFO:teuthology.orchestra.run.smithi154.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-06-06T08:19:37.912 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:37.912 INFO:teuthology.orchestra.run.smithi154.stdout:Total 10 MB/s | 27 MB 00:02 2024-06-06T08:19:38.027 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:19:38.063 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:19:38.063 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:19:38.423 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:19:38.424 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:19:38.973 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:19:39.026 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-06T08:19:39.058 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-06T08:19:39.091 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-06T08:19:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-06T08:19:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-06T08:19:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-06T08:19:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-06T08:19:39.375 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-06T08:19:39.409 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-06T08:19:39.442 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-06T08:19:39.483 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-06T08:19:39.544 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-06T08:19:39.551 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:19:39.586 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-06T08:19:39.602 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-06T08:19:39.624 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-06T08:19:39.634 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-06T08:19:39.668 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-06T08:19:39.698 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-06T08:19:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-06T08:19:39.844 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-06T08:19:39.846 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-06T08:19:39.886 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-06T08:19:39.915 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-06T08:19:39.921 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-06T08:19:39.952 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-06T08:19:39.969 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-06T08:19:39.986 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-06T08:19:40.007 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-06T08:19:40.022 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-06T08:19:40.062 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-06T08:19:40.064 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-06T08:19:40.098 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-06T08:19:40.123 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-06T08:19:40.132 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-06T08:19:40.161 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-06T08:19:40.236 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-06T08:19:40.325 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-06T08:19:40.370 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-06T08:19:40.402 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-06T08:19:40.456 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-06T08:19:40.487 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-06T08:19:40.540 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-06T08:19:40.544 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-06T08:19:40.577 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-06T08:19:40.578 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-06T08:19:40.627 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-06T08:19:40.692 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-06T08:19:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-06T08:19:40.841 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.n 28/33 2024-06-06T08:19:40.871 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 29/33 2024-06-06T08:19:40.890 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-06T08:19:40.897 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 29/33 2024-06-06T08:19:40.897 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:40.897 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-06T08:19:40.897 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T08:19:40.897 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T08:19:40.897 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:40.932 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-06T08:19:41.012 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-06T08:19:41.099 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-06T08:19:41.140 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-06T08:19:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-06T08:19:41.403 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.n 28/33 2024-06-06T08:19:41.434 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 29/33 2024-06-06T08:19:41.460 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 29/33 2024-06-06T08:19:41.460 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:41.460 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-06T08:19:41.460 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T08:19:41.460 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T08:19:41.460 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:42.213 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 30/33 2024-06-06T08:19:42.241 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 30/33 2024-06-06T08:19:42.241 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:42.241 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-06T08:19:42.241 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T08:19:42.241 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T08:19:42.241 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:42.625 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 31/33 2024-06-06T08:19:42.648 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 31/33 2024-06-06T08:19:42.648 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:42.648 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-06T08:19:42.649 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T08:19:42.649 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T08:19:42.649 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:42.785 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 30/33 2024-06-06T08:19:42.812 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 30/33 2024-06-06T08:19:42.812 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:42.812 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-06T08:19:42.813 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T08:19:42.813 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T08:19:42.813 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:42.850 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 32/33 2024-06-06T08:19:42.874 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 32/33 2024-06-06T08:19:42.874 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:42.874 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-06T08:19:42.874 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T08:19:42.874 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T08:19:42.875 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:42.939 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:17.2.7-992.g6ee68927.el9.x86_64 33/33 2024-06-06T08:19:43.186 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 31/33 2024-06-06T08:19:43.209 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 31/33 2024-06-06T08:19:43.210 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:43.210 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-06T08:19:43.210 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T08:19:43.210 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T08:19:43.210 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:43.411 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 32/33 2024-06-06T08:19:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 32/33 2024-06-06T08:19:43.435 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:43.435 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-06T08:19:43.435 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T08:19:43.435 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T08:19:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:17.2.7-992.g6ee68927.el9.x86_64 33/33 2024-06-06T08:19:44.654 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:17.2.7-992.g6ee68927.el9.x86_64 33/33 2024-06-06T08:19:44.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:17.2.7-992.g6ee68927.el9.x86_64 1/33 2024-06-06T08:19:44.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 2/33 2024-06-06T08:19:44.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 3/33 2024-06-06T08:19:44.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 4/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 5/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.n 6/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-06T08:19:44.655 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-06T08:19:44.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-06T08:19:44.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-06T08:19:44.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-06T08:19:44.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-06T08:19:44.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-06T08:19:44.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-06T08:19:44.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-06T08:19:44.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-06T08:19:44.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-06T08:19:44.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-06T08:19:44.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-06T08:19:44.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-06T08:19:44.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-06T08:19:44.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-06T08:19:44.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-06T08:19:44.658 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-06T08:19:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-2:17.2.7-992.g6ee68927.el9.x86_64 33/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:17.2.7-992.g6ee68927.el9.x86_64 1/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 2/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 3/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 4/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 5/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.n 6/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-06T08:19:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-06T08:19:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-06T08:19:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-06T08:19:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-06T08:19:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-06T08:19:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-06T08:19:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-06T08:19:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-06T08:19:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-06T08:19:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-06T08:19:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-06T08:19:45.509 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-06T08:19:45.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-06T08:19:45.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-06T08:19:45.511 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:45.512 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:19:45.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-06-06T08:19:46.247 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-06T08:19:46.248 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:46.248 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:19:46.248 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:46.248 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:46.248 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:46.248 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:46.249 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:46.249 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:46.249 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-06T08:19:46.249 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-06T08:19:46.249 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-06T08:19:46.249 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-06T08:19:46.249 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-06T08:19:46.249 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-06T08:19:46.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-06T08:19:46.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-06T08:19:46.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-06T08:19:46.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-06T08:19:46.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-06T08:19:46.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-06T08:19:46.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-06T08:19:46.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-06T08:19:46.250 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-06T08:19:46.251 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:46.252 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:19:46.456 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:55 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:46.537 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:19:46.579 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:19:46.580 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-06T08:19:46.580 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:19:46.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base 2024-06-06T08:19:46.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-06-06T08:19:47.204 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:56 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:47.227 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:56 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:47.286 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:19:47.328 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:19:47.330 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-06-06T08:19:47.330 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:19:47.350 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:19:47.351 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:47.351 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-06T08:19:47.351 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:47.351 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:19:47.351 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 82 k 2024-06-06T08:19:47.351 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:47.351 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:19:47.352 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:47.352 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-06T08:19:47.352 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:47.352 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 82 k 2024-06-06T08:19:47.352 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 372 k 2024-06-06T08:19:47.352 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:19:47.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install cephadm 2024-06-06T08:19:47.666 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-17.2.7-992.g6ee68927.el9.noarch.rpm 261 kB/s | 82 kB 00:00 2024-06-06T08:19:47.666 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:47.667 INFO:teuthology.orchestra.run.smithi086.stdout:Total 260 kB/s | 82 kB 00:00 2024-06-06T08:19:47.667 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:19:47.672 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:19:47.672 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:19:47.674 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:19:47.674 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:19:47.741 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:19:47.972 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:56 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:48.096 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:19:48.097 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:48.097 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-06-06T08:19:48.097 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:48.097 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:19:48.097 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 82 k 2024-06-06T08:19:48.097 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:48.097 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:19:48.097 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:48.098 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-06-06T08:19:48.098 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:48.098 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 82 k 2024-06-06T08:19:48.098 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 372 k 2024-06-06T08:19:48.098 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:19:48.157 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T08:19:48.253 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T08:19:48.411 INFO:teuthology.orchestra.run.smithi154.stdout:cephadm-17.2.7-992.g6ee68927.el9.noarch.rpm 262 kB/s | 82 kB 00:00 2024-06-06T08:19:48.411 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:48.412 INFO:teuthology.orchestra.run.smithi154.stdout:Total 261 kB/s | 82 kB 00:00 2024-06-06T08:19:48.412 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:19:48.417 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:19:48.417 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:19:48.419 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:19:48.419 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:19:48.487 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:19:48.597 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T08:19:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T08:19:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:48.861 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:19:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:48.862 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:19:48.943 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T08:19:49.040 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T08:19:49.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-06-06T08:19:49.460 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T08:19:49.654 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:58 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:49.700 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T08:19:49.700 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:49.700 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:19:49.701 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:49.701 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:49.701 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:19:49.783 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:19:49.783 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:49.783 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-06-06T08:19:49.783 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-992.g6ee68927.el9 ceph 138 k 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 138 k 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 379 k 2024-06-06T08:19:49.784 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:19:49.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-immutable-object-cache 2024-06-06T08:19:50.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-17.2.7-992.g6ee6892 419 kB/s | 138 kB 00:00 2024-06-06T08:19:50.114 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:50.114 INFO:teuthology.orchestra.run.smithi086.stdout:Total 417 kB/s | 138 kB 00:00 2024-06-06T08:19:50.114 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:19:50.122 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:19:50.122 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:19:50.179 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:19:50.180 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:19:50.321 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:19:50.381 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T08:19:50.406 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T08:19:50.406 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:50.406 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-06T08:19:50.406 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:50.522 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:59 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:50.649 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:19:50.650 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:50.650 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-06-06T08:19:50.650 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:50.650 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:19:50.651 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-992.g6ee68927.el9 ceph 138 k 2024-06-06T08:19:50.651 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:50.651 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:19:50.651 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:50.651 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-06-06T08:19:50.651 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:50.651 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 138 k 2024-06-06T08:19:50.651 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 379 k 2024-06-06T08:19:50.651 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:19:50.885 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-immutable-object-cache-17.2.7-992.g6ee6892 590 kB/s | 138 kB 00:00 2024-06-06T08:19:50.885 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:50.886 INFO:teuthology.orchestra.run.smithi154.stdout:Total 586 kB/s | 138 kB 00:00 2024-06-06T08:19:50.886 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:19:50.893 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:19:50.893 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:19:50.950 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:19:50.950 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:19:51.101 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:19:51.203 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T08:19:51.228 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T08:19:51.228 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:19:51.228 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-06T08:19:51.228 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:51.294 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T08:19:51.294 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:51.294 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:19:51.295 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:51.295 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:51.295 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:19:51.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-06-06T08:19:52.129 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:01 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:52.149 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T08:19:52.149 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:52.149 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:19:52.149 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:19:52.149 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:52.149 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:19:52.210 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:19:52.251 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:19:52.252 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-06T08:19:52.253 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:19:52.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-06-06T08:19:52.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr 2024-06-06T08:19:52.889 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:01 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:52.993 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:01 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:53.012 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:19:53.013 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:53.013 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-06T08:19:53.014 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:53.014 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:19:53.014 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 3.4 M 2024-06-06T08:19:53.014 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-06T08:19:53.014 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 24 k 2024-06-06T08:19:53.014 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 15 k 2024-06-06T08:19:53.014 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-06T08:19:53.014 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-06T08:19:53.014 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-06T08:19:53.015 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-06-06T08:19:53.015 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-06T08:19:53.015 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:53.015 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:19:53.015 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:19:53.015 INFO:teuthology.orchestra.run.smithi086.stdout:Install 7 Packages 2024-06-06T08:19:53.015 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:53.016 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.7 M 2024-06-06T08:19:53.016 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 80 M 2024-06-06T08:19:53.016 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:19:53.079 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:19:53.123 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:19:53.123 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-06-06T08:19:53.124 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:19:53.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-dashboard 2024-06-06T08:19:53.368 INFO:teuthology.orchestra.run.smithi086.stdout:(1/7): ceph-prometheus-alerts-17.2.7-992.g6ee68 52 kB/s | 15 kB 00:00 2024-06-06T08:19:53.410 INFO:teuthology.orchestra.run.smithi086.stdout:(2/7): ceph-grafana-dashboards-17.2.7-992.g6ee6 76 kB/s | 24 kB 00:00 2024-06-06T08:19:53.452 INFO:teuthology.orchestra.run.smithi086.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 108 kB/s | 9.0 kB 00:00 2024-06-06T08:19:53.477 INFO:teuthology.orchestra.run.smithi086.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-06-06T08:19:53.493 INFO:teuthology.orchestra.run.smithi086.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 501 kB/s | 41 kB 00:00 2024-06-06T08:19:53.552 INFO:teuthology.orchestra.run.smithi086.stdout:(6/7): ceph-mgr-dashboard-17.2.7-992.g6ee68927. 7.2 MB/s | 3.4 MB 00:00 2024-06-06T08:19:53.577 INFO:teuthology.orchestra.run.smithi086.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-06-06T08:19:53.581 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:53.581 INFO:teuthology.orchestra.run.smithi086.stdout:Total 6.4 MB/s | 3.7 MB 00:00 2024-06-06T08:19:53.581 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:19:53.589 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:19:53.589 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:19:53.651 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:19:53.651 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:19:53.752 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:02 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:53.876 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:19:53.877 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:53.877 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-06-06T08:19:53.877 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:53.878 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:19:53.878 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 3.4 M 2024-06-06T08:19:53.878 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-06-06T08:19:53.878 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 24 k 2024-06-06T08:19:53.878 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 15 k 2024-06-06T08:19:53.878 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-06T08:19:53.878 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-06T08:19:53.878 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-06T08:19:53.878 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-06-06T08:19:53.879 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-06T08:19:53.879 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:53.879 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:19:53.879 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:19:53.879 INFO:teuthology.orchestra.run.smithi154.stdout:Install 7 Packages 2024-06-06T08:19:53.879 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:53.880 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.7 M 2024-06-06T08:19:53.880 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 80 M 2024-06-06T08:19:53.880 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:19:53.956 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:19:54.004 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-06T08:19:54.038 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-06T08:19:54.085 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-06T08:19:54.114 INFO:teuthology.orchestra.run.smithi154.stdout:(1/7): ceph-grafana-dashboards-17.2.7-992.g6ee6 126 kB/s | 24 kB 00:00 2024-06-06T08:19:54.116 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-06T08:19:54.139 INFO:teuthology.orchestra.run.smithi154.stdout:(2/7): ceph-prometheus-alerts-17.2.7-992.g6ee68 66 kB/s | 15 kB 00:00 2024-06-06T08:19:54.145 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.no 5/7 2024-06-06T08:19:54.232 INFO:teuthology.orchestra.run.smithi154.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 77 kB/s | 9.0 kB 00:00 2024-06-06T08:19:54.266 INFO:teuthology.orchestra.run.smithi154.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 979 kB/s | 31 kB 00:00 2024-06-06T08:19:54.291 INFO:teuthology.orchestra.run.smithi154.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 271 kB/s | 41 kB 00:00 2024-06-06T08:19:54.316 INFO:teuthology.orchestra.run.smithi154.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-06-06T08:19:54.425 INFO:teuthology.orchestra.run.smithi154.stdout:(7/7): ceph-mgr-dashboard-17.2.7-992.g6ee68927. 6.6 MB/s | 3.4 MB 00:00 2024-06-06T08:19:54.428 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:54.428 INFO:teuthology.orchestra.run.smithi154.stdout:Total 6.6 MB/s | 3.7 MB 00:00 2024-06-06T08:19:54.429 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:19:54.437 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:19:54.437 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:19:54.499 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:19:54.500 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:19:54.796 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:19:54.867 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-06T08:19:54.902 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-06T08:19:54.950 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-06T08:19:54.980 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-06T08:19:55.009 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.no 5/7 2024-06-06T08:19:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.n 6/7 2024-06-06T08:19:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 7/7 2024-06-06T08:19:55.993 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.n 6/7 2024-06-06T08:19:56.029 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 7/7 2024-06-06T08:19:56.204 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 7/7 2024-06-06T08:19:56.205 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.n 1/7 2024-06-06T08:19:56.205 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 2/7 2024-06-06T08:19:56.205 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.no 3/7 2024-06-06T08:19:56.205 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-06T08:19:56.205 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-06T08:19:56.205 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-06T08:19:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-06T08:19:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:56.533 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:19:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:56.533 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-06T08:19:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-06T08:19:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-06T08:19:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-06T08:19:56.534 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:56.534 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:19:56.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-06T08:19:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 7/7 2024-06-06T08:19:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.n 1/7 2024-06-06T08:19:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 2/7 2024-06-06T08:19:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.no 3/7 2024-06-06T08:19:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-06T08:19:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-06T08:19:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-06T08:19:57.287 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:06 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:57.348 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-06T08:19:57.348 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:57.349 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:19:57.411 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 7.4 M 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-06T08:19:57.412 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-06-06T08:19:57.413 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:19:57.414 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 38 M 2024-06-06T08:19:57.414 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 209 M 2024-06-06T08:19:57.414 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:19:57.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-06T08:19:57.804 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:57.804 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:57.961 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 825 kB/s | 195 kB 00:00 2024-06-06T08:19:57.962 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:58.028 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-06-06T08:19:58.053 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 359 kB/s | 33 kB 00:00 2024-06-06T08:19:58.053 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:58.054 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:58.111 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 312 kB/s | 18 kB 00:00 2024-06-06T08:19:58.114 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:07 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:19:58.162 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-06-06T08:19:58.222 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-06-06T08:19:58.238 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-06-06T08:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-06-06T08:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-06-06T08:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 7.4 M 2024-06-06T08:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-06-06T08:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:19:58.240 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-06-06T08:19:58.241 INFO:teuthology.orchestra.run.smithi154.stdout:Install 12 Packages 2024-06-06T08:19:58.241 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:19:58.241 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 38 M 2024-06-06T08:19:58.241 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 209 M 2024-06-06T08:19:58.242 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:19:58.262 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-06-06T08:19:58.354 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-06-06T08:19:58.421 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 5.1 MB 00:00 2024-06-06T08:19:58.447 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.7 MB/s | 250 kB 00:00 2024-06-06T08:19:58.627 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:58.628 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:58.763 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-06-06T08:19:58.766 INFO:teuthology.orchestra.run.smithi154.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 903 kB/s | 195 kB 00:00 2024-06-06T08:19:58.767 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:58.816 INFO:teuthology.orchestra.run.smithi154.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2024-06-06T08:19:58.817 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:58.858 INFO:teuthology.orchestra.run.smithi154.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-06-06T08:19:58.859 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:19:58.942 INFO:teuthology.orchestra.run.smithi154.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 218 kB/s | 18 kB 00:00 2024-06-06T08:19:59.009 INFO:teuthology.orchestra.run.smithi154.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-06-06T08:19:59.034 INFO:teuthology.orchestra.run.smithi154.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 383 kB/s | 35 kB 00:00 2024-06-06T08:19:59.089 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-9 5.4 MB/s | 7.4 MB 00:01 2024-06-06T08:19:59.091 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:59.091 INFO:teuthology.orchestra.run.smithi086.stdout:Total 23 MB/s | 38 MB 00:01 2024-06-06T08:19:59.101 INFO:teuthology.orchestra.run.smithi154.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-06-06T08:19:59.185 INFO:teuthology.orchestra.run.smithi154.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-06-06T08:19:59.252 INFO:teuthology.orchestra.run.smithi154.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 250 kB 00:00 2024-06-06T08:19:59.319 INFO:teuthology.orchestra.run.smithi154.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-06-06T08:19:59.443 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:19:59.470 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:19:59.471 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:19:59.636 INFO:teuthology.orchestra.run.smithi154.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-06-06T08:19:59.719 INFO:teuthology.orchestra.run.smithi154.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-9 6.3 MB/s | 7.4 MB 00:01 2024-06-06T08:19:59.721 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:19:59.721 INFO:teuthology.orchestra.run.smithi154.stdout:Total 26 MB/s | 38 MB 00:01 2024-06-06T08:19:59.962 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:19:59.963 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:20:00.071 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:20:00.099 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:20:00.099 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:20:00.616 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:20:00.616 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:20:00.717 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:20:00.791 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-06T08:20:00.828 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-06T08:20:00.928 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-06T08:20:00.971 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-06T08:20:01.156 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-06T08:20:01.187 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-06T08:20:01.274 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-06T08:20:01.383 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:20:01.448 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-06T08:20:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-06T08:20:01.591 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-06T08:20:01.627 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-06T08:20:01.817 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-06T08:20:01.851 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-06T08:20:01.942 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-06T08:20:01.945 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-06T08:20:02.067 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-06T08:20:02.612 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-06T08:20:02.738 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-06T08:20:03.304 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T08:20:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T08:20:04.336 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-06T08:20:04.361 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 12/12 2024-06-06T08:20:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-06T08:20:05.051 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 12/12 2024-06-06T08:20:06.907 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 12/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 1/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T08:20:06.908 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-06T08:20:07.327 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-06T08:20:07.328 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:07.328 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:20:07.328 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:20:07.328 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-06T08:20:07.328 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-06T08:20:07.328 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-06T08:20:07.328 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-06T08:20:07.328 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-06T08:20:07.330 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-06T08:20:07.330 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-06T08:20:07.330 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-06T08:20:07.330 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-06T08:20:07.330 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-06T08:20:07.331 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-06T08:20:07.331 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:07.331 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:07.575 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 12/12 2024-06-06T08:20:07.575 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 1/12 2024-06-06T08:20:07.575 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-06T08:20:07.575 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-06T08:20:07.575 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-06T08:20:07.575 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-06T08:20:07.575 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-06T08:20:07.575 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-06T08:20:07.576 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-06T08:20:07.576 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-06T08:20:07.576 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T08:20:07.576 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-06T08:20:07.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-06-06T08:20:07.963 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-06T08:20:07.964 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-06T08:20:07.965 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-06T08:20:07.965 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-06T08:20:07.965 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-06T08:20:07.965 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-06T08:20:07.965 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:07.965 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:08.286 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:17 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:08.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-rook 2024-06-06T08:20:08.410 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:08.412 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:08.412 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-06T08:20:08.412 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:08.412 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:20:08.412 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 47 k 2024-06-06T08:20:08.412 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-06T08:20:08.412 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-06T08:20:08.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-06T08:20:08.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-06T08:20:08.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-06T08:20:08.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-06T08:20:08.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-06T08:20:08.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-06T08:20:08.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-06T08:20:08.414 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-06T08:20:08.414 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-06T08:20:08.414 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-06T08:20:08.414 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-06T08:20:08.414 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:08.414 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:20:08.414 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:08.415 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-06-06T08:20:08.415 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:08.415 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.2 M 2024-06-06T08:20:08.415 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-06-06T08:20:08.416 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:20:08.569 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:08.599 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:08.687 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 150 kB/s | 26 kB 00:00 2024-06-06T08:20:08.712 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 98 kB/s | 19 kB 00:00 2024-06-06T08:20:08.712 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:08.715 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-pyasn1-0.4.8-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.4.8-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:08.804 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-06-06T08:20:08.829 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-06-06T08:20:08.871 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 279 kB 00:00 2024-06-06T08:20:08.897 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 801 kB/s | 54 kB 00:00 2024-06-06T08:20:08.922 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): ceph-mgr-rook-17.2.7-992.g6ee68927.el9. 117 kB/s | 47 kB 00:00 2024-06-06T08:20:08.938 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:17 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:08.947 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 297 kB/s | 15 kB 00:00 2024-06-06T08:20:09.006 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-06-06T08:20:09.040 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.9 MB/s | 59 kB 00:00 2024-06-06T08:20:09.061 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 47 k 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-06T08:20:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:20:09.064 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:09.065 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2024-06-06T08:20:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:09.065 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.6 MB/s | 90 kB 00:00 2024-06-06T08:20:09.066 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.2 M 2024-06-06T08:20:09.066 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 26 M 2024-06-06T08:20:09.066 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:20:09.090 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-google-auth-2.29.0-1.el9.noarc 1.2 MB/s | 206 kB 00:00 2024-06-06T08:20:09.131 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-06-06T08:20:09.135 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:09.136 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-06-06T08:20:09.199 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:20:09.214 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:20:09.214 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:20:09.298 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:20:09.300 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:20:09.488 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:09.489 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:09.562 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 167 kB/s | 26 kB 00:00 2024-06-06T08:20:09.588 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 19 kB 00:00 2024-06-06T08:20:09.588 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:09.588 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-pyasn1-0.4.8-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.4.8-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:09.654 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-06-06T08:20:09.679 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-06-06T08:20:09.713 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.6 MB/s | 279 kB 00:00 2024-06-06T08:20:09.738 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-06-06T08:20:09.789 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): ceph-mgr-rook-17.2.7-992.g6ee68927.el9. 125 kB/s | 47 kB 00:00 2024-06-06T08:20:09.797 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:20:09.814 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-06-06T08:20:09.865 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 213 kB/s | 32 kB 00:00 2024-06-06T08:20:09.890 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-06-06T08:20:09.923 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-06-06T08:20:09.949 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): python3-google-auth-2.29.0-1.el9.noarc 1.3 MB/s | 206 kB 00:00 2024-06-06T08:20:09.955 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-06T08:20:09.991 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-06-06T08:20:09.994 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:09.994 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-06-06T08:20:10.008 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-06T08:20:10.059 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:20:10.064 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-06T08:20:10.074 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:20:10.074 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:20:10.117 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-06T08:20:10.159 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-06T08:20:10.159 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:20:10.160 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:20:10.251 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-06T08:20:10.359 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-06T08:20:10.408 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-06T08:20:10.661 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:20:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-06T08:20:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-06T08:20:10.963 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-06T08:20:11.000 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-06T08:20:11.041 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-06T08:20:11.122 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-06T08:20:11.135 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-06T08:20:11.179 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-06T08:20:11.213 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-06T08:20:11.244 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-06T08:20:11.259 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-06T08:20:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 13/13 2024-06-06T08:20:11.292 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-06T08:20:12.015 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-06T08:20:12.037 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 13/13 2024-06-06T08:20:12.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 1/13 2024-06-06T08:20:12.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-06T08:20:12.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-06T08:20:12.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-06T08:20:12.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-06T08:20:12.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-06T08:20:12.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-06T08:20:12.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-06T08:20:12.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-06T08:20:12.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-06T08:20:12.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-06T08:20:12.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-06T08:20:12.065 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-06T08:20:12.098 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-06T08:20:12.146 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-06T08:20:12.168 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 13/13 2024-06-06T08:20:12.351 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-06T08:20:12.351 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:12.351 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:20:12.351 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:20:12.351 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-06T08:20:12.351 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-06T08:20:12.351 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-06T08:20:12.351 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-06T08:20:12.351 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-06T08:20:12.352 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-06T08:20:12.352 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-06T08:20:12.352 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-06T08:20:12.352 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-06T08:20:12.352 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-06T08:20:12.352 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-06T08:20:12.352 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-06T08:20:12.352 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:12.352 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:12.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-06-06T08:20:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 13/13 2024-06-06T08:20:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 1/13 2024-06-06T08:20:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-06T08:20:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-06T08:20:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-06T08:20:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-06T08:20:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-06T08:20:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-06T08:20:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-06T08:20:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-06T08:20:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-06T08:20:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-06T08:20:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-06T08:20:13.171 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:22 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:13.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-06T08:20:13.262 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:13.262 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:20:13.262 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-06T08:20:13.263 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:13.264 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:13.295 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 123 k 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-06T08:20:13.296 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-06T08:20:13.297 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-06T08:20:13.297 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:13.297 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:20:13.297 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:13.297 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-06-06T08:20:13.297 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:13.297 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 7.0 M 2024-06-06T08:20:13.298 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 33 M 2024-06-06T08:20:13.298 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:20:13.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-cephadm 2024-06-06T08:20:13.615 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:13.617 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:13.772 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-06-06T08:20:13.797 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-mgr-cephadm-17.2.7-992.g6ee68927.el 474 kB/s | 123 kB 00:00 2024-06-06T08:20:13.822 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-06T08:20:13.847 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-06-06T08:20:13.873 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.3 MB/s | 548 kB 00:00 2024-06-06T08:20:14.014 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-06-06T08:20:14.017 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:14.017 INFO:teuthology.orchestra.run.smithi086.stdout:Total 9.7 MB/s | 7.0 MB 00:00 2024-06-06T08:20:14.087 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:20:14.098 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:20:14.099 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:20:14.099 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:23 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:14.194 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:20:14.195 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:20:14.223 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:14.224 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-06-06T08:20:14.224 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:14.224 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:20:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 123 k 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2024-06-06T08:20:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:14.226 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 7.0 M 2024-06-06T08:20:14.226 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 33 M 2024-06-06T08:20:14.226 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:20:14.502 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:20:14.552 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:14.553 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T08:20:14.672 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-06T08:20:14.715 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-06-06T08:20:14.724 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-06T08:20:14.749 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 16 MB/s | 548 kB 00:00 2024-06-06T08:20:14.775 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-06-06T08:20:14.808 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.5 MB/s | 51 kB 00:00 2024-06-06T08:20:14.833 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): ceph-mgr-cephadm-17.2.7-992.g6ee68927.el 344 kB/s | 123 kB 00:00 2024-06-06T08:20:14.959 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-06-06T08:20:14.960 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:14.961 INFO:teuthology.orchestra.run.smithi154.stdout:Total 9.5 MB/s | 7.0 MB 00:00 2024-06-06T08:20:15.031 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:20:15.042 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:20:15.042 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:20:15.140 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:20:15.140 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:20:15.245 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-06T08:20:15.327 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-06T08:20:15.383 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-06T08:20:15.420 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 6/6 2024-06-06T08:20:15.447 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:20:15.617 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-06T08:20:15.677 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-06T08:20:16.198 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 6/6 2024-06-06T08:20:16.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/6 2024-06-06T08:20:16.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-06T08:20:16.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-06T08:20:16.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-06T08:20:16.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-06T08:20:16.199 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-06T08:20:16.297 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-06T08:20:16.351 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-06T08:20:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 6/6 2024-06-06T08:20:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:16.481 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:16.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-06-06T08:20:17.140 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 6/6 2024-06-06T08:20:17.140 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/6 2024-06-06T08:20:17.140 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-06T08:20:17.140 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-06T08:20:17.140 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-06T08:20:17.140 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-06T08:20:17.300 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:26 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:17.397 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:17.398 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:17.422 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:17.422 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:17.422 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-06T08:20:17.422 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:17.422 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:20:17.422 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 813 k 2024-06-06T08:20:17.423 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-06T08:20:17.423 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-06T08:20:17.423 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:17.423 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:20:17.423 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:17.423 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-06-06T08:20:17.423 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:17.424 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 893 k 2024-06-06T08:20:17.424 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.6 M 2024-06-06T08:20:17.424 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:20:17.578 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:20:17.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-fuse 2024-06-06T08:20:17.708 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 486 kB/s | 80 kB 00:00 2024-06-06T08:20:17.958 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-17.2.7-992.g6ee68927.el9.x86_6 1.9 MB/s | 813 kB 00:00 2024-06-06T08:20:17.959 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:17.959 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.6 MB/s | 893 kB 00:00 2024-06-06T08:20:17.971 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:20:17.994 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:20:17.994 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:20:18.074 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:20:18.074 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:20:18.215 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:27 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:18.254 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:20:18.343 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:18.344 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-06-06T08:20:18.344 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:18.344 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:20:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 813 k 2024-06-06T08:20:18.344 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-06-06T08:20:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-06T08:20:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:18.345 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:20:18.345 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:18.345 INFO:teuthology.orchestra.run.smithi154.stdout:Install 2 Packages 2024-06-06T08:20:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:18.345 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 893 k 2024-06-06T08:20:18.346 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.6 M 2024-06-06T08:20:18.346 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:20:18.395 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-06T08:20:18.465 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2/2 2024-06-06T08:20:18.665 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T08:20:18.777 INFO:teuthology.orchestra.run.smithi154.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 488 kB/s | 80 kB 00:00 2024-06-06T08:20:18.910 INFO:teuthology.orchestra.run.smithi154.stdout:(2/2): ceph-fuse-17.2.7-992.g6ee68927.el9.x86_6 2.7 MB/s | 813 kB 00:00 2024-06-06T08:20:18.912 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:18.912 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.5 MB/s | 893 kB 00:00 2024-06-06T08:20:18.924 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:20:18.946 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:20:18.946 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:20:19.024 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:20:19.024 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:20:19.081 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2/2 2024-06-06T08:20:19.081 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/2 2024-06-06T08:20:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:20:19.331 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-06T08:20:19.355 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-06T08:20:19.355 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:19.355 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:20:19.356 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-06T08:20:19.356 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:19.356 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:19.410 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2/2 2024-06-06T08:20:19.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-06-06T08:20:20.040 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2/2 2024-06-06T08:20:20.041 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/2 2024-06-06T08:20:20.227 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:29 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-06T08:20:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:20.314 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:20:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-06T08:20:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:20.314 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:20.353 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:17.2.7-992.g6ee68927.el9 ceph 126 k 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 126 k 2024-06-06T08:20:20.354 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 448 k 2024-06-06T08:20:20.355 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:20:20.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados-devel 2024-06-06T08:20:20.611 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-17.2.7-992.g6ee68927.el9.x86_64. 493 kB/s | 126 kB 00:00 2024-06-06T08:20:20.612 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:20.612 INFO:teuthology.orchestra.run.smithi086.stdout:Total 490 kB/s | 126 kB 00:00 2024-06-06T08:20:20.612 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:20:20.617 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:20:20.617 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:20:20.678 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:20:20.679 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:20:20.830 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:20:20.933 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:21.138 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:30 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:21.266 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:21.267 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:21.267 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:17.2.7-992.g6ee68927.el9 ceph 126 k 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 126 k 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 448 k 2024-06-06T08:20:21.268 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:20:21.365 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:21.574 INFO:teuthology.orchestra.run.smithi154.stdout:librados-devel-17.2.7-992.g6ee68927.el9.x86_64. 414 kB/s | 126 kB 00:00 2024-06-06T08:20:21.575 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:21.575 INFO:teuthology.orchestra.run.smithi154.stdout:Total 412 kB/s | 126 kB 00:00 2024-06-06T08:20:21.575 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:20:21.580 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:20:21.580 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:20:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:21.638 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:20:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:21.638 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:21.643 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:20:21.644 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:20:21.818 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:20:21.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-06-06T08:20:21.922 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:22.431 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:31 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:22.513 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:22.555 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:22.556 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-06T08:20:22.556 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:22.587 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:22.587 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:22.588 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:20:22.588 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:22.588 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:22.588 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:22.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-06-06T08:20:22.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs2 2024-06-06T08:20:23.218 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:32 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:23.341 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:23.342 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:23.343 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-06-06T08:20:23.343 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:23.343 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:20:23.343 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:17.2.7-992.g6ee68927.el9 ceph 25 k 2024-06-06T08:20:23.343 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:23.344 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:20:23.344 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:23.344 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-06T08:20:23.344 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:23.344 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 25 k 2024-06-06T08:20:23.344 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 110 k 2024-06-06T08:20:23.345 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:20:23.399 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:32 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:23.484 INFO:teuthology.orchestra.run.smithi154.stdout:Package libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:23.506 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-17.2.7-992.g6ee68927.el9.x86_64 153 kB/s | 25 kB 00:00 2024-06-06T08:20:23.506 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:23.507 INFO:teuthology.orchestra.run.smithi086.stdout:Total 152 kB/s | 25 kB 00:00 2024-06-06T08:20:23.507 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:20:23.510 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:20:23.510 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:20:23.512 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:20:23.512 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:20:23.528 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:23.529 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-06-06T08:20:23.529 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:20:23.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs-devel 2024-06-06T08:20:23.695 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:24.176 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:33 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:24.271 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:24.271 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:24.271 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:20:24.271 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:24.271 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:24.271 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:24.301 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:24.301 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:24.301 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-06-06T08:20:24.302 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:24.302 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:20:24.302 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:17.2.7-992.g6ee68927.el9 ceph 25 k 2024-06-06T08:20:24.302 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:24.302 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:20:24.302 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:24.302 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-06-06T08:20:24.302 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:24.303 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 25 k 2024-06-06T08:20:24.303 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 110 k 2024-06-06T08:20:24.303 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:20:24.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-06-06T08:20:24.544 INFO:teuthology.orchestra.run.smithi154.stdout:libcephfs-devel-17.2.7-992.g6ee68927.el9.x86_64 103 kB/s | 25 kB 00:00 2024-06-06T08:20:24.544 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:24.544 INFO:teuthology.orchestra.run.smithi154.stdout:Total 103 kB/s | 25 kB 00:00 2024-06-06T08:20:24.545 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:20:24.548 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:20:24.548 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:20:24.550 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:20:24.550 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:20:24.630 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:20:24.724 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:25.019 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:34 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:25.049 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:25.100 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:25.141 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:25.142 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-06T08:20:25.142 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:25.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-06-06T08:20:25.267 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:25.268 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:25.268 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:20:25.268 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:25.268 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:25.268 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:25.462 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados2 2024-06-06T08:20:25.771 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:34 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:25.852 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:25.893 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:25.894 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-06T08:20:25.894 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:25.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-06-06T08:20:26.033 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:35 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:26.115 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:26.156 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:26.157 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-06-06T08:20:26.157 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:26.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librbd1 2024-06-06T08:20:26.528 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:35 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:26.609 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:26.650 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:26.651 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-06T08:20:26.651 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:26.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-06-06T08:20:26.795 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:35 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:26.876 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:26.917 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:26.918 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-06-06T08:20:26.918 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:26.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rados 2024-06-06T08:20:27.287 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:36 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:27.369 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:27.411 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:27.412 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-06T08:20:27.412 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:27.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-06-06T08:20:27.564 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:36 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:27.645 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:27.687 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:27.688 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-06-06T08:20:27.688 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:27.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rgw 2024-06-06T08:20:28.060 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:37 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:28.144 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:28.187 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:28.188 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-06T08:20:28.188 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:28.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-06-06T08:20:28.327 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:37 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:28.410 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:28.451 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:28.452 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-06-06T08:20:28.452 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:28.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-cephfs 2024-06-06T08:20:28.833 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:37 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:28.914 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:28.956 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:28.957 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-06T08:20:28.957 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:29.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-06-06T08:20:29.092 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:38 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:29.175 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:29.219 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:29.220 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-06-06T08:20:29.220 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:29.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rbd 2024-06-06T08:20:29.603 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:38 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:29.724 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:29.724 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 82 k 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 82 k 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 222 k 2024-06-06T08:20:29.725 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:20:29.861 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:38 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:29.947 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T08:20:29.950 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-17.2.7-992.g6ee68927.el9.x86_64.rpm 363 kB/s | 82 kB 00:00 2024-06-06T08:20:29.951 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:29.951 INFO:teuthology.orchestra.run.smithi086.stdout:Total 361 kB/s | 82 kB 00:00 2024-06-06T08:20:29.952 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:20:29.958 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:20:29.958 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:20:29.985 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:29.986 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-06-06T08:20:29.986 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:30.021 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:20:30.021 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:20:30.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-fuse 2024-06-06T08:20:30.164 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:20:30.297 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:30.625 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:39 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:30.647 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:30.747 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:30.748 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:30.748 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-06-06T08:20:30.748 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:30.748 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:20:30.748 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 82 k 2024-06-06T08:20:30.748 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:30.748 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:20:30.748 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:30.749 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-06-06T08:20:30.749 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:30.749 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 82 k 2024-06-06T08:20:30.749 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 222 k 2024-06-06T08:20:30.749 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:20:30.871 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:30.871 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:30.871 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:20:30.871 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:30.871 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:30.871 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:30.972 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-fuse-17.2.7-992.g6ee68927.el9.x86_64.rpm 367 kB/s | 82 kB 00:00 2024-06-06T08:20:30.973 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:30.973 INFO:teuthology.orchestra.run.smithi154.stdout:Total 364 kB/s | 82 kB 00:00 2024-06-06T08:20:30.973 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:20:30.980 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:20:30.980 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:20:31.043 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:20:31.043 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:20:31.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-06-06T08:20:31.184 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:20:31.293 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:31.653 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:40 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:31.676 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:31.779 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:31.780 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:31.780 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-06-06T08:20:31.780 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:31.780 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:20:31.780 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.9 M 2024-06-06T08:20:31.780 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:31.780 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:20:31.780 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:31.781 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-06T08:20:31.781 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:31.781 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.9 M 2024-06-06T08:20:31.781 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 12 M 2024-06-06T08:20:31.781 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:20:31.936 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:31.936 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:31.936 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:20:31.936 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:31.936 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:31.936 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:32.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-mirror 2024-06-06T08:20:32.686 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-17.2.7-992.g6ee68927.el9.x86_64.rpm 3.2 MB/s | 2.9 MB 00:00 2024-06-06T08:20:32.687 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:32.687 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.2 MB/s | 2.9 MB 00:00 2024-06-06T08:20:32.687 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:20:32.696 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:20:32.697 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:20:32.741 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:41 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:32.777 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:20:32.778 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:20:32.869 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:32.869 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.9 M 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.9 M 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 12 M 2024-06-06T08:20:32.870 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:20:33.204 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:20:33.270 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:33.295 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:33.295 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:20:33.296 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-06T08:20:33.296 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T08:20:33.296 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T08:20:33.296 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:33.442 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-mirror-17.2.7-992.g6ee68927.el9.x86_64.rpm 5.1 MB/s | 2.9 MB 00:00 2024-06-06T08:20:33.443 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:33.443 INFO:teuthology.orchestra.run.smithi154.stdout:Total 5.1 MB/s | 2.9 MB 00:00 2024-06-06T08:20:33.443 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:20:33.453 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:20:33.453 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:20:33.534 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:20:33.535 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:20:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:20:33.992 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:34.017 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T08:20:34.017 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-06T08:20:34.017 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T08:20:34.017 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T08:20:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:34.343 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:34.343 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:34.343 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:20:34.343 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:34.343 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:34.343 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:34.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-06-06T08:20:35.092 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:35.092 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:35.092 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:20:35.092 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:35.092 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:35.092 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:35.176 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:44 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:35.303 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-06T08:20:35.304 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:35.304 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-06-06T08:20:35.305 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:35.305 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-06T08:20:35.305 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 165 k 2024-06-06T08:20:35.305 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:35.305 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-06T08:20:35.305 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-06T08:20:35.305 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-06T08:20:35.305 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:35.305 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 165 k 2024-06-06T08:20:35.306 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 477 k 2024-06-06T08:20:35.306 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-06T08:20:35.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-nbd 2024-06-06T08:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-17.2.7-992.g6ee68927.el9.x86_64.rpm 471 kB/s | 165 kB 00:00 2024-06-06T08:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout:Total 467 kB/s | 165 kB 00:00 2024-06-06T08:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-06T08:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-06T08:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-06T08:20:35.732 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-06T08:20:35.733 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-06T08:20:35.886 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-06T08:20:35.894 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:44 ago on Thu 06 Jun 2024 08:18:51 AM UTC. 2024-06-06T08:20:35.972 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:36.018 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-06-06T08:20:36.019 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 165 k 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 165 k 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 477 k 2024-06-06T08:20:36.020 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-06-06T08:20:36.370 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-nbd-17.2.7-992.g6ee68927.el9.x86_64.rpm 471 kB/s | 165 kB 00:00 2024-06-06T08:20:36.371 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:20:36.371 INFO:teuthology.orchestra.run.smithi154.stdout:Total 469 kB/s | 165 kB 00:00 2024-06-06T08:20:36.371 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-06-06T08:20:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:36.379 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-06-06T08:20:36.379 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-06-06T08:20:36.444 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-06-06T08:20:36.444 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-06-06T08:20:36.596 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-06-06T08:20:36.613 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:36.613 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:36.613 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-06T08:20:36.614 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:36.614 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:20:36.614 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-06T08:20:36.694 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:36.863 DEBUG:teuthology.parallel:result is None 2024-06-06T08:20:37.111 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T08:20:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:37.379 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-06-06T08:20:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T08:20:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:20:37.379 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-06-06T08:20:37.692 DEBUG:teuthology.parallel:result is None 2024-06-06T08:20:37.693 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T08:20:37.693 INFO:teuthology.packaging:ref: None 2024-06-06T08:20:37.693 INFO:teuthology.packaging:tag: None 2024-06-06T08:20:37.693 INFO:teuthology.packaging:branch: quincy 2024-06-06T08:20:37.693 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T08:20:37.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-06T08:20:37.912 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-06T08:20:37.938 INFO:teuthology.orchestra.run.smithi086.stdout:17.2.7-992.g6ee68927.el9 2024-06-06T08:20:37.939 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-992.g6ee68927.el9 2024-06-06T08:20:37.939 INFO:teuthology.task.install:The correct ceph version 17.2.7-992.g6ee68927 is installed. 2024-06-06T08:20:37.940 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T08:20:37.941 INFO:teuthology.packaging:ref: None 2024-06-06T08:20:37.941 INFO:teuthology.packaging:tag: None 2024-06-06T08:20:37.941 INFO:teuthology.packaging:branch: quincy 2024-06-06T08:20:37.941 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T08:20:37.941 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-06T08:20:38.270 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-06T08:20:38.294 INFO:teuthology.orchestra.run.smithi154.stdout:17.2.7-992.g6ee68927.el9 2024-06-06T08:20:38.294 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-992.g6ee68927.el9 2024-06-06T08:20:38.294 INFO:teuthology.task.install:The correct ceph version 17.2.7-992.g6ee68927 is installed. 2024-06-06T08:20:38.296 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-06T08:20:38.296 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:20:38.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-06T08:20:38.330 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:20:38.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-06T08:20:38.387 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-06T08:20:38.388 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:20:38.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-06-06T08:20:38.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-06T08:20:38.486 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:20:38.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-06-06T08:20:38.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-06T08:20:38.577 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-06T08:20:38.577 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:20:38.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-06T08:20:38.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-06T08:20:38.673 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:20:38.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-06T08:20:38.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-06T08:20:38.764 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-06T08:20:38.764 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:20:38.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-06-06T08:20:38.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-06T08:20:38.858 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:20:38.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-06-06T08:20:38.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-06T08:20:38.950 INFO:teuthology.run_tasks:Running task print... 2024-06-06T08:20:38.959 INFO:teuthology.task.print:**** done install task... 2024-06-06T08:20:38.959 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-06T08:20:39.066 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-06T08:20:39.066 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-06-06T08:20:39.066 INFO:tasks.cephadm:Cluster fsid is a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:20:39.066 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-06T08:20:39.066 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-06-06T08:20:39.067 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi086': '172.21.15.86', 'mon.smithi154': '172.21.15.154'} 2024-06-06T08:20:39.067 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-06T08:20:39.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-06-06T08:20:39.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2024-06-06T08:20:39.119 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-06-06T08:20:39.119 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-06T08:20:39.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 6 08:20 /home/ubuntu/cephtest/cephadm 2024-06-06T08:20:39.382 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-06T08:20:39.589 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 6 08:20 /home/ubuntu/cephtest/cephadm 2024-06-06T08:20:39.589 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-06T08:20:39.607 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-06T08:20:39.633 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-06-06T08:20:39.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-06T08:20:39.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-06T08:20:40.536 INFO:teuthology.orchestra.run.smithi154.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-06T08:20:40.561 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-06T08:21:09.917 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-06T08:21:09.917 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable)", 2024-06-06T08:21:09.917 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a", 2024-06-06T08:21:09.917 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-06-06T08:21:09.917 INFO:teuthology.orchestra.run.smithi086.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a" 2024-06-06T08:21:09.917 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-06T08:21:09.917 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-06T08:21:15.606 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-06-06T08:21:15.606 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_version": "ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable)", 2024-06-06T08:21:15.606 INFO:teuthology.orchestra.run.smithi154.stdout: "image_id": "7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a", 2024-06-06T08:21:15.606 INFO:teuthology.orchestra.run.smithi154.stdout: "repo_digests": [ 2024-06-06T08:21:15.606 INFO:teuthology.orchestra.run.smithi154.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a" 2024-06-06T08:21:15.606 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-06-06T08:21:15.606 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-06-06T08:21:15.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-06-06T08:21:15.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2024-06-06T08:21:15.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-06-06T08:21:15.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2024-06-06T08:21:15.751 INFO:tasks.cephadm:Writing seed config... 2024-06-06T08:21:15.753 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-06T08:21:15.754 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-06T08:21:15.754 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-06-06T08:21:15.754 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-06-06T08:21:15.754 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-06-06T08:21:15.754 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-06T08:21:15.754 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-06T08:21:15.754 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-06-06T08:21:15.755 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-06-06T08:21:15.755 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-06T08:21:15.755 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-06T08:21:15.755 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-06-06T08:21:15.755 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-06T08:21:15.755 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-06T08:21:15.755 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-06T08:21:15.756 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-06T08:21:15.757 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-06T08:21:15.757 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-06T08:21:15.757 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-06T08:21:15.757 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-06T08:21:15.757 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-06T08:21:15.757 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-06T08:21:15.757 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-06T08:21:15.757 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-06T08:21:15.758 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-06T08:21:15.758 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-06T08:21:15.758 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-06T08:21:15.758 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-06T08:21:15.758 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-06T08:21:15.758 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-06T08:21:15.759 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:21:15.759 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-06T08:21:15.784 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a8550dee-23dd-11ef-bc9b-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-06T08:21:15.785 DEBUG:teuthology.orchestra.run.smithi086:mon.smithi086> sudo journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi086.service 2024-06-06T08:21:15.827 INFO:tasks.cephadm:Bootstrapping... 2024-06-06T08:21:15.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.86 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-06T08:21:16.035 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-06T08:21:16.035 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'a8550dee-23dd-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.86', '--skip-admin-label'] 2024-06-06T08:21:16.060 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 5.0.2 2024-06-06T08:21:16.060 INFO:teuthology.orchestra.run.smithi086.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-06T08:21:16.061 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying podman|docker is present... 2024-06-06T08:21:16.084 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 5.0.2 2024-06-06T08:21:16.084 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying lvm2 is present... 2024-06-06T08:21:16.084 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying time synchronization is in place... 2024-06-06T08:21:16.092 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-06T08:21:16.092 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-06T08:21:16.098 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-06T08:21:16.098 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-06T08:21:16.105 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout enabled 2024-06-06T08:21:16.112 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout active 2024-06-06T08:21:16.112 INFO:teuthology.orchestra.run.smithi086.stdout:Unit chronyd.service is enabled and running 2024-06-06T08:21:16.112 INFO:teuthology.orchestra.run.smithi086.stdout:Repeating the final host check... 2024-06-06T08:21:16.136 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 5.0.2 2024-06-06T08:21:16.136 INFO:teuthology.orchestra.run.smithi086.stdout:podman (/bin/podman) version 5.0.2 is present 2024-06-06T08:21:16.136 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl is present 2024-06-06T08:21:16.136 INFO:teuthology.orchestra.run.smithi086.stdout:lvcreate is present 2024-06-06T08:21:16.143 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-06T08:21:16.143 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-06T08:21:16.149 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-06T08:21:16.149 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-06T08:21:16.156 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout enabled 2024-06-06T08:21:16.162 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout active 2024-06-06T08:21:16.163 INFO:teuthology.orchestra.run.smithi086.stdout:Unit chronyd.service is enabled and running 2024-06-06T08:21:16.163 INFO:teuthology.orchestra.run.smithi086.stdout:Host looks OK 2024-06-06T08:21:16.163 INFO:teuthology.orchestra.run.smithi086.stdout:Cluster fsid: a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:21:16.163 INFO:teuthology.orchestra.run.smithi086.stdout:Acquiring lock 140262400111040 on /run/cephadm/a8550dee-23dd-11ef-bc9b-c7b262605968.lock 2024-06-06T08:21:16.163 INFO:teuthology.orchestra.run.smithi086.stdout:Lock 140262400111040 acquired on /run/cephadm/a8550dee-23dd-11ef-bc9b-c7b262605968.lock 2024-06-06T08:21:16.164 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying IP 172.21.15.86 port 3300 ... 2024-06-06T08:21:16.164 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying IP 172.21.15.86 port 6789 ... 2024-06-06T08:21:16.164 INFO:teuthology.orchestra.run.smithi086.stdout:Base mon IP(s) is [172.21.15.86:3300, 172.21.15.86:6789], mon addrv is [v2:172.21.15.86:3300,v1:172.21.15.86:6789] 2024-06-06T08:21:16.167 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.86 metric 100 2024-06-06T08:21:16.167 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.86 metric 100 2024-06-06T08:21:16.170 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-06T08:21:16.170 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-06T08:21:16.170 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2024-06-06T08:21:16.173 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-06T08:21:16.173 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-06T08:21:16.173 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-06T08:21:16.173 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-06T08:21:16.173 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d809/64 scope link 2024-06-06T08:21:16.174 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-06T08:21:16.174 INFO:teuthology.orchestra.run.smithi086.stdout:Mon IP `172.21.15.86` is in CIDR network `172.21.0.0/20` 2024-06-06T08:21:16.174 INFO:teuthology.orchestra.run.smithi086.stdout:Mon IP `172.21.15.86` is in CIDR network `172.21.0.0/20` 2024-06-06T08:21:16.174 INFO:teuthology.orchestra.run.smithi086.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-06T08:21:16.175 INFO:teuthology.orchestra.run.smithi086.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-06T08:21:16.175 INFO:teuthology.orchestra.run.smithi086.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-06T08:21:16.641 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a 2024-06-06T08:21:16.642 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-06T08:21:16.642 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Getting image source signatures 2024-06-06T08:21:16.642 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Copying blob sha256:36850128dbaa55a7fd38bf12640a70ab57c121fdc76af02f72b423e059979691 2024-06-06T08:21:16.642 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Copying blob sha256:63e3eeadb2c456ff44649abe69523acc2aac9d8a9fe03158716a851e0cecadea 2024-06-06T08:21:16.642 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Copying config sha256:7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a 2024-06-06T08:21:16.642 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-06T08:21:17.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph: stdout ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable) 2024-06-06T08:21:17.530 INFO:teuthology.orchestra.run.smithi086.stdout:Ceph version: ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable) 2024-06-06T08:21:17.530 INFO:teuthology.orchestra.run.smithi086.stdout:Extracting ceph user uid/gid from container image... 2024-06-06T08:21:18.501 INFO:teuthology.orchestra.run.smithi086.stdout:stat: stdout 167 167 2024-06-06T08:21:18.501 INFO:teuthology.orchestra.run.smithi086.stdout:Creating initial keys... 2024-06-06T08:21:19.382 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph-authtool: stdout AQB+cWFmM0KbMBAAVijWX47aRpBfIf45gQgY6A== 2024-06-06T08:21:20.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph-authtool: stdout AQB/cWFmiOb+KBAAYhcOkMcO7Ci7bkRuqx3Llg== 2024-06-06T08:21:21.188 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph-authtool: stdout AQCAcWFm0ntpHhAAtPg6Nhd1+d1h3SNmSa7p5g== 2024-06-06T08:21:21.188 INFO:teuthology.orchestra.run.smithi086.stdout:Creating initial monmap... 2024-06-06T08:21:22.156 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-06T08:21:22.156 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-06-06T08:21:22.156 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:21:22.156 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-06T08:21:22.156 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool for smithi086 [v2:172.21.15.86:3300,v1:172.21.15.86:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-06T08:21:22.157 INFO:teuthology.orchestra.run.smithi086.stdout:setting min_mon_release = octopus 2024-06-06T08:21:22.157 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: set fsid to a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:21:22.157 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-06T08:21:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:21:22.157 INFO:teuthology.orchestra.run.smithi086.stdout:Creating mon... 2024-06-06T08:21:23.131 INFO:teuthology.orchestra.run.smithi086.stdout:create mon.smithi086 on 2024-06-06T08:21:23.353 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-06T08:21:23.496 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-06T08:21:23.638 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968.target. 2024-06-06T08:21:23.639 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968.target. 2024-06-06T08:21:23.804 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi086 2024-06-06T08:21:23.804 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi086.service: Unit ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi086.service not loaded. 2024-06-06T08:21:23.958 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968.target.wants/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi086.service → /etc/systemd/system/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@.service. 2024-06-06T08:21:24.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:24 smithi086 podman[28704]: 2024-06-06 08:21:24.097844446 +0000 UTC m=+0.017709075 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph:quincy 2024-06-06T08:21:25.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:24 smithi086 podman[28704]: 2024-06-06 08:21:24.980308264 +0000 UTC m=+0.900172875 container create cfecebf0777bedabeb01d9c88aaa6a862b83e504175dcd59d7cc417c5e5dd0de (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-6ee6892, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, CEPH_POINT_RELEASE=) 2024-06-06T08:21:27.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:26 smithi086 podman[28704]: 2024-06-06 08:21:26.840355704 +0000 UTC m=+2.760220328 container init cfecebf0777bedabeb01d9c88aaa6a862b83e504175dcd59d7cc417c5e5dd0de (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=) 2024-06-06T08:21:27.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:26 smithi086 podman[28704]: 2024-06-06 08:21:26.844900989 +0000 UTC m=+2.764765606 container start cfecebf0777bedabeb01d9c88aaa6a862b83e504175dcd59d7cc417c5e5dd0de (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_CLEAN=True, RELEASE=quincy-6ee6892, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-06T08:21:27.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:26 smithi086 ceph-mon[28736]: mkfs a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:21:27.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:26 smithi086 ceph-mon[28736]: mon.smithi086 is new leader, mons smithi086 in quorum (ranks 0) 2024-06-06T08:21:27.778 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T08:21:27.778 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-06-06T08:21:27.787 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T08:21:27.787 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-06T08:21:27.787 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-06-06T08:21:27.787 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to enable service . firewalld.service is not available 2024-06-06T08:21:27.787 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mon to start... 2024-06-06T08:21:27.788 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mon... 2024-06-06T08:21:28.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:27 smithi086 bash[28704]: cfecebf0777bedabeb01d9c88aaa6a862b83e504175dcd59d7cc417c5e5dd0de 2024-06-06T08:21:28.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:27 smithi086 systemd[1]: Started Ceph mon.smithi086 for a8550dee-23dd-11ef-bc9b-c7b262605968. 2024-06-06T08:21:28.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:27 smithi086 ceph-mon[28736]: mon.smithi086 is new leader, mons smithi086 in quorum (ranks 0) 2024-06-06T08:21:28.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:27 smithi086 ceph-mon[28736]: monmap e1: 1 mons at {smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T08:21:28.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:27 smithi086 ceph-mon[28736]: fsmap 2024-06-06T08:21:28.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:27 smithi086 ceph-mon[28736]: osdmap e1: 0 total, 0 up, 0 in 2024-06-06T08:21:28.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:27 smithi086 ceph-mon[28736]: mgrmap e1: no daemons active 2024-06-06T08:21:30.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:30 smithi086 ceph-mon[28736]: from='client.? 172.21.15.86:0/1299453661' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-06T08:21:36.454 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout cluster: 2024-06-06T08:21:36.456 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout id: a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:21:36.456 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-06T08:21:36.456 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:36.456 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout services: 2024-06-06T08:21:36.456 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi086 (age 3s) 2024-06-06T08:21:36.456 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-06T08:21:36.456 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-06T08:21:36.456 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:36.456 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout data: 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout pgs: 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.508+0000 7fef6b9d4640 1 Processor -- start 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.508+0000 7fef6b9d4640 1 -- start start 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.509+0000 7fef6b9d4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64104940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.509+0000 7fef6b9d4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef64104e80 con 0x7fef64075ba0 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.509+0000 7fef69770640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64104940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:36.457 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.509+0000 7fef69770640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64104940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52796/0 (socket says 172.21.15.86:52796) 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.509+0000 7fef69770640 1 -- 172.21.15.86:0/2418352397 learned_addr learned my addr 172.21.15.86:0/2418352397 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.509+0000 7fef69770640 1 -- 172.21.15.86:0/2418352397 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef64104fc0 con 0x7fef64075ba0 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.511+0000 7fef69770640 1 --2- 172.21.15.86:0/2418352397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64104940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fef60009840 tx=0x7fef600091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a098860678c3059 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.511+0000 7fef5bfff640 1 -- 172.21.15.86:0/2418352397 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef60032440 con 0x7fef64075ba0 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.512+0000 7fef5bfff640 1 -- 172.21.15.86:0/2418352397 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef600325a0 con 0x7fef64075ba0 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.512+0000 7fef5bfff640 1 -- 172.21.15.86:0/2418352397 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef600328d0 con 0x7fef64075ba0 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.512+0000 7fef6b9d4640 1 -- 172.21.15.86:0/2418352397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 msgr2=0x7fef64104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.512+0000 7fef6b9d4640 1 --2- 172.21.15.86:0/2418352397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64104940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fef60009840 tx=0x7fef600091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.512+0000 7fef6b9d4640 1 -- 172.21.15.86:0/2418352397 shutdown_connections 2024-06-06T08:21:36.458 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.512+0000 7fef6b9d4640 1 --2- 172.21.15.86:0/2418352397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64104940 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.512+0000 7fef6b9d4640 1 -- 172.21.15.86:0/2418352397 >> 172.21.15.86:0/2418352397 conn(0x7fef640fc430 msgr2=0x7fef640fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.512+0000 7fef6b9d4640 1 -- 172.21.15.86:0/2418352397 shutdown_connections 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.512+0000 7fef6b9d4640 1 -- 172.21.15.86:0/2418352397 wait complete. 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.513+0000 7fef6b9d4640 1 Processor -- start 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.513+0000 7fef6b9d4640 1 -- start start 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.513+0000 7fef6b9d4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64110360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.513+0000 7fef6b9d4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef641108a0 con 0x7fef64075ba0 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.513+0000 7fef69770640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64110360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.513+0000 7fef69770640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64110360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52810/0 (socket says 172.21.15.86:52810) 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.513+0000 7fef69770640 1 -- 172.21.15.86:0/1299453661 learned_addr learned my addr 172.21.15.86:0/1299453661 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:36.459 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.513+0000 7fef69770640 1 -- 172.21.15.86:0/1299453661 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef60009450 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.514+0000 7fef69770640 1 --2- 172.21.15.86:0/1299453661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64110360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fef6002a700 tx=0x7fef60032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.514+0000 7fef59ffb640 1 -- 172.21.15.86:0/1299453661 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef60031be0 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.514+0000 7fef59ffb640 1 -- 172.21.15.86:0/1299453661 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef60031d40 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.514+0000 7fef6b9d4640 1 -- 172.21.15.86:0/1299453661 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef64110aa0 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.514+0000 7fef59ffb640 1 -- 172.21.15.86:0/1299453661 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef6003b5a0 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.514+0000 7fef6b9d4640 1 -- 172.21.15.86:0/1299453661 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef64110eb0 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.514+0000 7fef59ffb640 1 -- 172.21.15.86:0/1299453661 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fef60039040 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.514+0000 7fef59ffb640 1 -- 172.21.15.86:0/1299453661 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fef60041030 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.515+0000 7fef6b9d4640 1 -- 172.21.15.86:0/1299453661 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6404b480 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.517+0000 7fef59ffb640 1 -- 172.21.15.86:0/1299453661 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fef60037070 con 0x7fef64075ba0 2024-06-06T08:21:36.460 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.579+0000 7fef6b9d4640 1 -- 172.21.15.86:0/1299453661 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fef64109ac0 con 0x7fef64075ba0 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.579+0000 7fef59ffb640 1 -- 172.21.15.86:0/1299453661 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+318 (secure 0 0 0) 0x7fef6003bbd0 con 0x7fef64075ba0 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.580+0000 7fef6b9d4640 1 -- 172.21.15.86:0/1299453661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 msgr2=0x7fef64110360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.580+0000 7fef6b9d4640 1 --2- 172.21.15.86:0/1299453661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64110360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fef6002a700 tx=0x7fef60032000 comp rx=0 tx=0).stop 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.580+0000 7fef6b9d4640 1 -- 172.21.15.86:0/1299453661 shutdown_connections 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.580+0000 7fef6b9d4640 1 --2- 172.21.15.86:0/1299453661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef64075ba0 0x7fef64110360 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.580+0000 7fef6b9d4640 1 -- 172.21.15.86:0/1299453661 >> 172.21.15.86:0/1299453661 conn(0x7fef640fc430 msgr2=0x7fef640fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.580+0000 7fef6b9d4640 1 -- 172.21.15.86:0/1299453661 shutdown_connections 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:30.580+0000 7fef6b9d4640 1 -- 172.21.15.86:0/1299453661 wait complete. 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:mon is available 2024-06-06T08:21:36.461 INFO:teuthology.orchestra.run.smithi086.stdout:Assimilating anything we can from ceph.conf... 2024-06-06T08:21:37.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:36 smithi086 ceph-mon[28736]: from='client.? 172.21.15.86:0/744978011' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-06T08:21:37.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:36 smithi086 ceph-mon[28736]: from='client.? 172.21.15.86:0/744978011' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-06T08:21:37.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:37.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [global] 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout fsid = a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.86:3300,v1:172.21.15.86:6789] 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-06T08:21:37.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [osd] 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.863+0000 7fb58839d640 1 Processor -- start 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.863+0000 7fb58839d640 1 -- start start 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.864+0000 7fb58839d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb5801021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.864+0000 7fb58839d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb580102700 con 0x7fb580101db0 2024-06-06T08:21:37.521 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.864+0000 7fb586139640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb5801021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.864+0000 7fb586139640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb5801021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52818/0 (socket says 172.21.15.86:52818) 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.864+0000 7fb586139640 1 -- 172.21.15.86:0/1294029121 learned_addr learned my addr 172.21.15.86:0/1294029121 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.864+0000 7fb586139640 1 -- 172.21.15.86:0/1294029121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580102840 con 0x7fb580101db0 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.865+0000 7fb586139640 1 --2- 172.21.15.86:0/1294029121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb5801021c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb578009840 tx=0x7fb5780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb621fc600b5dc6 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.865+0000 7fb585137640 1 -- 172.21.15.86:0/1294029121 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb578032440 con 0x7fb580101db0 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.865+0000 7fb585137640 1 -- 172.21.15.86:0/1294029121 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5780325a0 con 0x7fb580101db0 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.865+0000 7fb585137640 1 -- 172.21.15.86:0/1294029121 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5780328d0 con 0x7fb580101db0 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.866+0000 7fb58839d640 1 -- 172.21.15.86:0/1294029121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 msgr2=0x7fb5801021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.866+0000 7fb58839d640 1 --2- 172.21.15.86:0/1294029121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb5801021c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb578009840 tx=0x7fb5780091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.866+0000 7fb58839d640 1 -- 172.21.15.86:0/1294029121 shutdown_connections 2024-06-06T08:21:37.522 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.866+0000 7fb58839d640 1 --2- 172.21.15.86:0/1294029121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb5801021c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.866+0000 7fb58839d640 1 -- 172.21.15.86:0/1294029121 >> 172.21.15.86:0/1294029121 conn(0x7fb5800fd5b0 msgr2=0x7fb5800ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.866+0000 7fb58839d640 1 -- 172.21.15.86:0/1294029121 shutdown_connections 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.866+0000 7fb58839d640 1 -- 172.21.15.86:0/1294029121 wait complete. 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.866+0000 7fb58839d640 1 Processor -- start 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.866+0000 7fb58839d640 1 -- start start 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb58839d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb580110350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb58839d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb580110890 con 0x7fb580101db0 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb586139640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb580110350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb586139640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb580110350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52832/0 (socket says 172.21.15.86:52832) 2024-06-06T08:21:37.523 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb586139640 1 -- 172.21.15.86:0/744978011 learned_addr learned my addr 172.21.15.86:0/744978011 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb586139640 1 -- 172.21.15.86:0/744978011 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb578009450 con 0x7fb580101db0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb586139640 1 --2- 172.21.15.86:0/744978011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb580110350 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb57802a700 tx=0x7fb578032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb576ffd640 1 -- 172.21.15.86:0/744978011 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb578031be0 con 0x7fb580101db0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb576ffd640 1 -- 172.21.15.86:0/744978011 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb578031d40 con 0x7fb580101db0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb576ffd640 1 -- 172.21.15.86:0/744978011 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb57803b5a0 con 0x7fb580101db0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb58839d640 1 -- 172.21.15.86:0/744978011 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb580110a90 con 0x7fb580101db0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.867+0000 7fb58839d640 1 -- 172.21.15.86:0/744978011 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb580110ea0 con 0x7fb580101db0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.868+0000 7fb576ffd640 1 -- 172.21.15.86:0/744978011 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fb578039040 con 0x7fb580101db0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.868+0000 7fb576ffd640 1 -- 172.21.15.86:0/744978011 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb578041030 con 0x7fb580101db0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.868+0000 7fb58839d640 1 -- 172.21.15.86:0/744978011 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58004b480 con 0x7fb580101db0 2024-06-06T08:21:37.524 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.871+0000 7fb576ffd640 1 -- 172.21.15.86:0/744978011 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fb578037070 con 0x7fb580101db0 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.929+0000 7fb58839d640 1 -- 172.21.15.86:0/744978011 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb580109af0 con 0x7fb580101db0 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.933+0000 7fb576ffd640 1 -- 172.21.15.86:0/744978011 <== mon.0 v2:172.21.15.86:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb578046d00 con 0x7fb580101db0 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.933+0000 7fb576ffd640 1 -- 172.21.15.86:0/744978011 <== mon.0 v2:172.21.15.86:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fb57804eca0 con 0x7fb580101db0 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.933+0000 7fb58839d640 1 -- 172.21.15.86:0/744978011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 msgr2=0x7fb580110350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.933+0000 7fb58839d640 1 --2- 172.21.15.86:0/744978011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb580110350 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb57802a700 tx=0x7fb578032000 comp rx=0 tx=0).stop 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.934+0000 7fb58839d640 1 -- 172.21.15.86:0/744978011 shutdown_connections 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.934+0000 7fb58839d640 1 --2- 172.21.15.86:0/744978011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb580101db0 0x7fb580110350 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.934+0000 7fb58839d640 1 -- 172.21.15.86:0/744978011 >> 172.21.15.86:0/744978011 conn(0x7fb5800fd5b0 msgr2=0x7fb580107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.934+0000 7fb58839d640 1 -- 172.21.15.86:0/744978011 shutdown_connections 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:36.934+0000 7fb58839d640 1 -- 172.21.15.86:0/744978011 wait complete. 2024-06-06T08:21:37.525 INFO:teuthology.orchestra.run.smithi086.stdout:Generating new minimal ceph.conf... 2024-06-06T08:21:38.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:37 smithi086 ceph-mon[28736]: from='client.? 172.21.15.86:0/2981696436' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:21:38.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.902+0000 7ff69a4f3640 1 Processor -- start 2024-06-06T08:21:38.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.903+0000 7ff69a4f3640 1 -- start start 2024-06-06T08:21:38.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.903+0000 7ff69a4f3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694103770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:38.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.903+0000 7ff69a4f3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940708a0 con 0x7ff694103360 2024-06-06T08:21:38.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.903+0000 7ff693fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694103770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:38.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.903+0000 7ff693fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694103770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59250/0 (socket says 172.21.15.86:59250) 2024-06-06T08:21:38.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.903+0000 7ff693fff640 1 -- 172.21.15.86:0/2123233951 learned_addr learned my addr 172.21.15.86:0/2123233951 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.903+0000 7ff693fff640 1 -- 172.21.15.86:0/2123233951 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff694103cb0 con 0x7ff694103360 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.904+0000 7ff693fff640 1 --2- 172.21.15.86:0/2123233951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694103770 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff684009840 tx=0x7ff6840091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b903b22062f396 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.904+0000 7ff692ffd640 1 -- 172.21.15.86:0/2123233951 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff684032440 con 0x7ff694103360 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.904+0000 7ff692ffd640 1 -- 172.21.15.86:0/2123233951 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff684032a30 con 0x7ff694103360 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.904+0000 7ff692ffd640 1 -- 172.21.15.86:0/2123233951 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff684031ad0 con 0x7ff694103360 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2123233951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 msgr2=0x7ff694103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 --2- 172.21.15.86:0/2123233951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694103770 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff684009840 tx=0x7ff6840091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2123233951 shutdown_connections 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 --2- 172.21.15.86:0/2123233951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694103770 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2123233951 >> 172.21.15.86:0/2123233951 conn(0x7ff6940fbc20 msgr2=0x7ff6940fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2123233951 shutdown_connections 2024-06-06T08:21:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2123233951 wait complete. 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 Processor -- start 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 -- start start 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694110aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.905+0000 7ff69a4f3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff694110fe0 con 0x7ff694103360 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff693fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694110aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff693fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694110aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59264/0 (socket says 172.21.15.86:59264) 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff693fff640 1 -- 172.21.15.86:0/2981696436 learned_addr learned my addr 172.21.15.86:0/2981696436 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff693fff640 1 -- 172.21.15.86:0/2981696436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff684009450 con 0x7ff694103360 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff693fff640 1 --2- 172.21.15.86:0/2981696436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694110aa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff684009420 tx=0x7ff684008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff690ff9640 1 -- 172.21.15.86:0/2981696436 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff68403d5b0 con 0x7ff694103360 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2981696436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6941111e0 con 0x7ff694103360 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff690ff9640 1 -- 172.21.15.86:0/2981696436 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff68403dba0 con 0x7ff694103360 2024-06-06T08:21:38.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff690ff9640 1 -- 172.21.15.86:0/2981696436 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff68403bb40 con 0x7ff694103360 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2981696436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff694111650 con 0x7ff694103360 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff690ff9640 1 -- 172.21.15.86:0/2981696436 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7ff684039070 con 0x7ff694103360 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.906+0000 7ff690ff9640 1 -- 172.21.15.86:0/2981696436 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff684046900 con 0x7ff694103360 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.907+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2981696436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69404b480 con 0x7ff694103360 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.909+0000 7ff690ff9640 1 -- 172.21.15.86:0/2981696436 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7ff694111650 con 0x7ff694103360 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.969+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2981696436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff694111d70 con 0x7ff694103360 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.969+0000 7ff690ff9640 1 -- 172.21.15.86:0/2981696436 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7ff684041020 con 0x7ff694103360 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.970+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2981696436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 msgr2=0x7ff694110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.970+0000 7ff69a4f3640 1 --2- 172.21.15.86:0/2981696436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694110aa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff684009420 tx=0x7ff684008040 comp rx=0 tx=0).stop 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.970+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2981696436 shutdown_connections 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.970+0000 7ff69a4f3640 1 --2- 172.21.15.86:0/2981696436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff694103360 0x7ff694110aa0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.970+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2981696436 >> 172.21.15.86:0/2981696436 conn(0x7ff6940fbc20 msgr2=0x7ff6940fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.970+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2981696436 shutdown_connections 2024-06-06T08:21:38.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:37.970+0000 7ff69a4f3640 1 -- 172.21.15.86:0/2981696436 wait complete. 2024-06-06T08:21:38.576 INFO:teuthology.orchestra.run.smithi086.stdout:Restarting the monitor... 2024-06-06T08:21:38.818 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:38 smithi086 systemd[1]: Stopping Ceph mon.smithi086 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:21:39.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:38 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086[28731]: 2024-06-06T08:21:38.819+0000 7f6c4d56e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi086 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T08:21:39.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:38 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086[28731]: 2024-06-06T08:21:38.819+0000 7f6c4d56e640 -1 mon.smithi086@0(leader) e1 *** Got Signal Terminated *** 2024-06-06T08:21:39.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:38 smithi086 podman[28979]: 2024-06-06 08:21:38.962951206 +0000 UTC m=+0.229342422 container died cfecebf0777bedabeb01d9c88aaa6a862b83e504175dcd59d7cc417c5e5dd0de (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-6ee6892, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, ceph=True, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS) 2024-06-06T08:21:39.349 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:39 smithi086 podman[28979]: 2024-06-06 08:21:39.332550842 +0000 UTC m=+0.598942048 container cleanup cfecebf0777bedabeb01d9c88aaa6a862b83e504175dcd59d7cc417c5e5dd0de (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-06T08:21:39.349 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:39 smithi086 bash[28979]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086 2024-06-06T08:21:39.974 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:39 smithi086 podman[28994]: 2024-06-06 08:21:39.603474345 +0000 UTC m=+0.641069701 container remove cfecebf0777bedabeb01d9c88aaa6a862b83e504175dcd59d7cc417c5e5dd0de (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-6ee6892, GIT_CLEAN=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-06T08:21:39.974 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:39 smithi086 systemd[1]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi086.service: Deactivated successfully. 2024-06-06T08:21:39.974 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:39 smithi086 systemd[1]: Stopped Ceph mon.smithi086 for a8550dee-23dd-11ef-bc9b-c7b262605968. 2024-06-06T08:21:39.975 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:39 smithi086 systemd[1]: Starting Ceph mon.smithi086 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:21:40.224 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:39 smithi086 podman[29094]: 2024-06-06 08:21:39.878316711 +0000 UTC m=+0.016829026 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph:quincy 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:39 smithi086 podman[29094]: 2024-06-06 08:21:39.979298367 +0000 UTC m=+0.117810656 container create 7a051db34bc21836df291f9223fbe26ec42d4a988dcdc6dbedfa4f4772b981c4 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 podman[29094]: 2024-06-06 08:21:40.197976366 +0000 UTC m=+0.336488664 container init 7a051db34bc21836df291f9223fbe26ec42d4a988dcdc6dbedfa4f4772b981c4 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, ceph=True, CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 podman[29094]: 2024-06-06 08:21:40.20110127 +0000 UTC m=+0.339613567 container start 7a051db34bc21836df291f9223fbe26ec42d4a988dcdc6dbedfa4f4772b981c4 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: set uid:gid to 167:167 (ceph:ceph) 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable), process ceph-mon, pid 2 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: pidfile_write: ignore empty --pid-file 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: load: jerasure load: lrc 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: RocksDB version: 6.15.5 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Compile date Jun 5 2024 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: DB SUMMARY 2024-06-06T08:21:40.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: DB Session ID: C19XMCVM0UUKBSGPYPHA 2024-06-06T08:21:40.226 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: CURRENT file: CURRENT 2024-06-06T08:21:40.226 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: IDENTITY file: IDENTITY 2024-06-06T08:21:40.226 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-06T08:21:40.226 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi086/store.db dir, Total Num: 1, files: 000008.sst 2024-06-06T08:21:40.226 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi086/store.db: 000010.log size: 98460 ; 2024-06-06T08:21:40.226 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.error_if_exists: 0 2024-06-06T08:21:40.226 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.create_if_missing: 0 2024-06-06T08:21:40.226 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.paranoid_checks: 1 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.env: 0x564030313440 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.fs: Posix File System 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.info_log: 0x564032a8f140 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_file_opening_threads: 16 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.statistics: (nil) 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.use_fsync: 0 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_log_file_size: 0 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-06T08:21:40.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.keep_log_file_num: 1000 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.recycle_log_file_num: 0 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.allow_fallocate: 1 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.allow_mmap_reads: 0 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.allow_mmap_writes: 0 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.use_direct_reads: 0 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.create_missing_column_families: 0 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.db_log_dir: 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi086/store.db 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-06T08:21:40.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.advise_random_on_open: 1 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.db_write_buffer_size: 0 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.write_buffer_manager: 0x564032b4e840 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.rate_limiter: (nil) 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.wal_recovery_mode: 2 2024-06-06T08:21:40.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.enable_thread_tracking: 0 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.enable_pipelined_write: 0 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.unordered_write: 0 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.row_cache: None 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.wal_filter: None 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-06T08:21:40.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.allow_ingest_behind: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.preserve_deletes: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.two_write_queues: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.manual_wal_flush: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.atomic_flush: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.log_readahead_size: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.best_efforts_recovery: 0 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-06T08:21:40.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.allow_data_in_errors: 0 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.db_host_id: __hostname__ 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_background_jobs: 2 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_background_compactions: -1 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_subcompactions: 1 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_total_wal_size: 0 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_open_files: -1 2024-06-06T08:21:40.232 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bytes_per_sync: 0 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_readahead_size: 0 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_background_flushes: -1 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Compression algorithms supported: 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: kZSTD supported: 0 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: kXpressCompression supported: 0 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: kBZip2Compression supported: 0 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: kLZ4Compression supported: 1 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: kZlibCompression supported: 1 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: kLZ4HCCompression supported: 1 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: kSnappyCompression supported: 1 2024-06-06T08:21:40.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000009 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.merge_operator: 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_filter: None 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_filter_factory: None 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.sst_partitioner_factory: None 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564032a37a68) 2024-06-06T08:21:40.234 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_type: 0 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_block_index_type: 0 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_shortening: 1 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: hash_index_allow_collision: 1 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: checksum: 1 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: no_block_cache: 0 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache: 0x564032a6d1f0 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-06-06T08:21:40.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_options: 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: capacity : 536870912 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: num_shard_bits : 4 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: strict_capacity_limit : 0 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_compressed: (nil) 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: persistent_cache: (nil) 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_size: 4096 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_size_deviation: 10 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_restart_interval: 16 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_block_restart_interval: 1 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: metadata_block_size: 4096 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: partition_filters: 0 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: use_delta_encoding: 1 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-06T08:21:40.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: whole_key_filtering: 1 2024-06-06T08:21:40.237 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: verify_compression: 0 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: format_version: 4 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: enable_index_compression: 1 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_align: 0 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.write_buffer_size: 33554432 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_write_buffer_number: 2 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compression: NoCompression 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bottommost_compression: Disabled 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.prefix_extractor: nullptr 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.num_levels: 7 2024-06-06T08:21:40.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compression_opts.level: 32767 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compression_opts.strategy: 0 2024-06-06T08:21:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compression_opts.enabled: false 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.target_file_size_base: 67108864 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-06T08:21:40.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.arena_block_size: 4194304 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.disable_auto_compactions: 0 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-06T08:21:40.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.table_properties_collectors: 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.inplace_update_support: 0 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-06T08:21:40.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.bloom_locality: 0 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.max_successive_merges: 0 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.paranoid_file_checks: 0 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.force_consistency_checks: 1 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.report_bg_io_stats: 0 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.ttl: 2592000 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.enable_blob_files: false 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.min_blob_size: 0 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.blob_file_size: 268435456 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-06T08:21:40.243 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717662100224299, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717662100225161, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93511, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91623, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13606, "raw_average_key_size": 49, "raw_value_size": 84161, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1717662100, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "ea163d9f-f643-4bb7-ad94-f0e3e2457c63", "db_session_id": "C19XMCVM0UUKBSGPYPHA"}} 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717662100225472, "job": 1, "event": "recovery_finished"} 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi086/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x564032aa0700 2024-06-06T08:21:40.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: DB pointer 0x564032a70000 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** DB Stats ** 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-06T08:21:40.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: L0 2/0 93.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.001 0 0 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Sum 2/0 93.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.001 0 0 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.001 0 0 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.0 0.00 0.00 1 0.001 0 0 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-06T08:21:40.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative compaction: 0.00 GB write, 28.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval compaction: 0.00 GB write, 28.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: L0 2/0 93.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.001 0 0 2024-06-06T08:21:40.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Sum 2/0 93.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.0 0.00 0.00 1 0.001 0 0 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.0 0.00 0.00 1 0.001 0 0 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative compaction: 0.00 GB write, 27.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T08:21:40.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: starting mon.smithi086 rank 0 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi086 fsid a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mon.smithi086@-1(???) e1 preinit fsid a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mon.smithi086@-1(???).mds e1 new map 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mon.smithi086@-1(???).mds e1 print_map 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: e1 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: legacy client fscid: -1 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: No filesystems configured 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mon.smithi086@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-06T08:21:40.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mon.smithi086@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-06T08:21:40.250 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mon.smithi086@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-06T08:21:40.250 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mon.smithi086@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-06T08:21:40.250 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mon.smithi086@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-06T08:21:40.267 INFO:teuthology.orchestra.run.smithi086.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-06T08:21:40.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 bash[29094]: 7a051db34bc21836df291f9223fbe26ec42d4a988dcdc6dbedfa4f4772b981c4 2024-06-06T08:21:40.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 systemd[1]: Started Ceph mon.smithi086 for a8550dee-23dd-11ef-bc9b-c7b262605968. 2024-06-06T08:21:40.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mon.smithi086 is new leader, mons smithi086 in quorum (ranks 0) 2024-06-06T08:21:40.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: monmap e1: 1 mons at {smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T08:21:40.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: fsmap 2024-06-06T08:21:40.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: osdmap e1: 0 total, 0 up, 0 in 2024-06-06T08:21:40.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:40 smithi086 ceph-mon[29114]: mgrmap e1: no daemons active 2024-06-06T08:21:41.288 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.660+0000 7fd0ec5b4640 1 Processor -- start 2024-06-06T08:21:41.293 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.661+0000 7fd0ec5b4640 1 -- start start 2024-06-06T08:21:41.293 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.661+0000 7fd0ec5b4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4103770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:41.293 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.661+0000 7fd0ec5b4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e40708a0 con 0x7fd0e4103360 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.661+0000 7fd0ea350640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4103770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.661+0000 7fd0ea350640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4103770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59274/0 (socket says 172.21.15.86:59274) 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.661+0000 7fd0ea350640 1 -- 172.21.15.86:0/2612735902 learned_addr learned my addr 172.21.15.86:0/2612735902 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.662+0000 7fd0ea350640 1 -- 172.21.15.86:0/2612735902 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e4103cb0 con 0x7fd0e4103360 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.664+0000 7fd0ea350640 1 --2- 172.21.15.86:0/2612735902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4103770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009840 tx=0x7fd0d80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fb4aac05d9f3f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.664+0000 7fd0e934e640 1 -- 172.21.15.86:0/2612735902 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d8032440 con 0x7fd0e4103360 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.664+0000 7fd0e934e640 1 -- 172.21.15.86:0/2612735902 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd0d8032a30 con 0x7fd0e4103360 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.664+0000 7fd0e934e640 1 -- 172.21.15.86:0/2612735902 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d8031ad0 con 0x7fd0e4103360 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.664+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/2612735902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 msgr2=0x7fd0e4103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.664+0000 7fd0ec5b4640 1 --2- 172.21.15.86:0/2612735902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4103770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009840 tx=0x7fd0d80091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.664+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/2612735902 shutdown_connections 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.664+0000 7fd0ec5b4640 1 --2- 172.21.15.86:0/2612735902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4103770 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:41.294 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.665+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/2612735902 >> 172.21.15.86:0/2612735902 conn(0x7fd0e40fbc20 msgr2=0x7fd0e40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.665+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/2612735902 shutdown_connections 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.665+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/2612735902 wait complete. 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.665+0000 7fd0ec5b4640 1 Processor -- start 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.665+0000 7fd0ec5b4640 1 -- start start 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.665+0000 7fd0ec5b4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4102a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.665+0000 7fd0ec5b4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e4102fb0 con 0x7fd0e4103360 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.665+0000 7fd0ea350640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0ea350640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4102a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59276/0 (socket says 172.21.15.86:59276) 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0ea350640 1 -- 172.21.15.86:0/4287232454 learned_addr learned my addr 172.21.15.86:0/4287232454 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0ea350640 1 -- 172.21.15.86:0/4287232454 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8009450 con 0x7fd0e4103360 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0ea350640 1 --2- 172.21.15.86:0/4287232454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4102a70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80317b0 tx=0x7fd0d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0d6ffd640 1 -- 172.21.15.86:0/4287232454 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d8031e90 con 0x7fd0e4103360 2024-06-06T08:21:41.295 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0d6ffd640 1 -- 172.21.15.86:0/4287232454 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd0d803daf0 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/4287232454 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e4101040 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0d6ffd640 1 -- 172.21.15.86:0/4287232454 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d803baa0 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/4287232454 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e41014b0 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0d6ffd640 1 -- 172.21.15.86:0/4287232454 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fd0d8039070 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.666+0000 7fd0d6ffd640 1 -- 172.21.15.86:0/4287232454 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd0d8046980 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.667+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/4287232454 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e404b480 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.669+0000 7fd0d6ffd640 1 -- 172.21.15.86:0/4287232454 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fd0d8041020 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.727+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/4287232454 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd0e4100820 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.732+0000 7fd0d6ffd640 1 -- 172.21.15.86:0/4287232454 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd0e4100820 con 0x7fd0e4103360 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.732+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/4287232454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 msgr2=0x7fd0e4102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.732+0000 7fd0ec5b4640 1 --2- 172.21.15.86:0/4287232454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4102a70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80317b0 tx=0x7fd0d8008040 comp rx=0 tx=0).stop 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.732+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/4287232454 shutdown_connections 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.732+0000 7fd0ec5b4640 1 --2- 172.21.15.86:0/4287232454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4103360 0x7fd0e4102a70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:41.296 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.732+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/4287232454 >> 172.21.15.86:0/4287232454 conn(0x7fd0e40fbc20 msgr2=0x7fd0e40fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:41.297 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.732+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/4287232454 shutdown_connections 2024-06-06T08:21:41.297 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:40.733+0000 7fd0ec5b4640 1 -- 172.21.15.86:0/4287232454 wait complete. 2024-06-06T08:21:41.297 INFO:teuthology.orchestra.run.smithi086.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-06T08:21:41.297 INFO:teuthology.orchestra.run.smithi086.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-06T08:21:41.297 INFO:teuthology.orchestra.run.smithi086.stdout:Creating mgr... 2024-06-06T08:21:41.297 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying port 9283 ... 2024-06-06T08:21:41.467 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mgr.smithi086.mfkgze 2024-06-06T08:21:41.467 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mgr.smithi086.mfkgze.service: Unit ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mgr.smithi086.mfkgze.service not loaded. 2024-06-06T08:21:41.695 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968.target.wants/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mgr.smithi086.mfkgze.service → /etc/systemd/system/ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@.service. 2024-06-06T08:21:42.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:41 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/4287232454' entity='client.admin' 2024-06-06T08:21:42.626 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T08:21:42.650 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-06-06T08:21:42.655 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T08:21:42.655 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-06T08:21:42.656 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-06-06T08:21:42.656 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to enable service . firewalld.service is not available 2024-06-06T08:21:42.656 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T08:21:42.656 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-06-06T08:21:42.657 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T08:21:42.657 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-06T08:21:42.658 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-06-06T08:21:42.658 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-06T08:21:42.658 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr to start... 2024-06-06T08:21:42.658 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr... 2024-06-06T08:21:43.441 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:43 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3497800992' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-06T08:21:43.730 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsid": "a8550dee-23dd-11ef-bc9b-c7b262605968", 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "health": { 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 0 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "smithi086" 2024-06-06T08:21:43.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-06T08:21:43.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:43.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "restful" 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-06T08:21:43.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modified": "2024-06-06T08:21:26.883287+0000", 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.049+0000 7f5843cff640 1 Processor -- start 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.049+0000 7f5843cff640 1 -- start start 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.049+0000 7f5843cff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c104940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.049+0000 7f5843cff640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c104e80 con 0x7f583c075ba0 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.049+0000 7f5841a9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c104940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.050+0000 7f5841a9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c104940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59306/0 (socket says 172.21.15.86:59306) 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.050+0000 7f5841a9b640 1 -- 172.21.15.86:0/412971111 learned_addr learned my addr 172.21.15.86:0/412971111 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:43.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.050+0000 7f5841a9b640 1 -- 172.21.15.86:0/412971111 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c104fc0 con 0x7f583c075ba0 2024-06-06T08:21:43.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.050+0000 7f5841a9b640 1 --2- 172.21.15.86:0/412971111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c104940 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5838009840 tx=0x7f58380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df02ec4b75c66591 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:43.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5840a99640 1 -- 172.21.15.86:0/412971111 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5838032440 con 0x7f583c075ba0 2024-06-06T08:21:43.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5840a99640 1 -- 172.21.15.86:0/412971111 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5838032a30 con 0x7f583c075ba0 2024-06-06T08:21:43.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5840a99640 1 -- 172.21.15.86:0/412971111 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5838031ad0 con 0x7f583c075ba0 2024-06-06T08:21:43.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5843cff640 1 -- 172.21.15.86:0/412971111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 msgr2=0x7f583c104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:43.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5843cff640 1 --2- 172.21.15.86:0/412971111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c104940 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5838009840 tx=0x7f58380091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:43.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5843cff640 1 -- 172.21.15.86:0/412971111 shutdown_connections 2024-06-06T08:21:43.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5843cff640 1 --2- 172.21.15.86:0/412971111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c104940 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:43.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5843cff640 1 -- 172.21.15.86:0/412971111 >> 172.21.15.86:0/412971111 conn(0x7f583c0fc430 msgr2=0x7f583c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5843cff640 1 -- 172.21.15.86:0/412971111 shutdown_connections 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.051+0000 7f5843cff640 1 -- 172.21.15.86:0/412971111 wait complete. 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.052+0000 7f5843cff640 1 Processor -- start 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.052+0000 7f5843cff640 1 -- start start 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.052+0000 7f5843cff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c112dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.052+0000 7f5843cff640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c113310 con 0x7f583c075ba0 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.052+0000 7f5841a9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c112dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.052+0000 7f5841a9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c112dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59320/0 (socket says 172.21.15.86:59320) 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.052+0000 7f5841a9b640 1 -- 172.21.15.86:0/3497800992 learned_addr learned my addr 172.21.15.86:0/3497800992 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.052+0000 7f5841a9b640 1 -- 172.21.15.86:0/3497800992 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838009450 con 0x7f583c075ba0 2024-06-06T08:21:43.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.053+0000 7f5841a9b640 1 --2- 172.21.15.86:0/3497800992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c112dd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5838031b90 tx=0x7f5838008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.053+0000 7f58327fc640 1 -- 172.21.15.86:0/3497800992 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58380328b0 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.053+0000 7f58327fc640 1 -- 172.21.15.86:0/3497800992 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f583803d8e0 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.053+0000 7f5843cff640 1 -- 172.21.15.86:0/3497800992 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c113510 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.053+0000 7f58327fc640 1 -- 172.21.15.86:0/3497800992 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f583803ba20 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.053+0000 7f5843cff640 1 -- 172.21.15.86:0/3497800992 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c113f60 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.053+0000 7f58327fc640 1 -- 172.21.15.86:0/3497800992 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f5838039070 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.053+0000 7f58327fc640 1 -- 172.21.15.86:0/3497800992 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f58380468e0 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.054+0000 7f5843cff640 1 -- 172.21.15.86:0/3497800992 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c05ead0 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.056+0000 7f58327fc640 1 -- 172.21.15.86:0/3497800992 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f5838065790 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.137+0000 7f5843cff640 1 -- 172.21.15.86:0/3497800992 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f583c105ce0 con 0x7f583c075ba0 2024-06-06T08:21:43.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.137+0000 7f58327fc640 1 -- 172.21.15.86:0/3497800992 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f5838054020 con 0x7f583c075ba0 2024-06-06T08:21:43.739 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.138+0000 7f5843cff640 1 -- 172.21.15.86:0/3497800992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 msgr2=0x7f583c112dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:43.739 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.138+0000 7f5843cff640 1 --2- 172.21.15.86:0/3497800992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c112dd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5838031b90 tx=0x7f5838008040 comp rx=0 tx=0).stop 2024-06-06T08:21:43.739 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.138+0000 7f5843cff640 1 -- 172.21.15.86:0/3497800992 shutdown_connections 2024-06-06T08:21:43.739 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.138+0000 7f5843cff640 1 --2- 172.21.15.86:0/3497800992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583c075ba0 0x7f583c112dd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:43.739 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.138+0000 7f5843cff640 1 -- 172.21.15.86:0/3497800992 >> 172.21.15.86:0/3497800992 conn(0x7f583c0fc430 msgr2=0x7f583c0fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:43.739 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.138+0000 7f5843cff640 1 -- 172.21.15.86:0/3497800992 shutdown_connections 2024-06-06T08:21:43.739 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:43.138+0000 7f5843cff640 1 -- 172.21.15.86:0/3497800992 wait complete. 2024-06-06T08:21:43.739 INFO:teuthology.orchestra.run.smithi086.stdout:mgr not available, waiting (1/15)... 2024-06-06T08:21:46.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:46 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1278202230' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-06T08:21:46.723 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:46.723 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-06T08:21:46.723 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsid": "a8550dee-23dd-11ef-bc9b-c7b262605968", 2024-06-06T08:21:46.723 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "health": { 2024-06-06T08:21:46.723 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-06T08:21:46.723 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-06T08:21:46.724 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-06T08:21:46.724 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:46.724 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-06T08:21:46.724 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-06T08:21:46.724 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 0 2024-06-06T08:21:46.724 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-06T08:21:46.724 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-06T08:21:46.724 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "smithi086" 2024-06-06T08:21:46.725 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-06T08:21:46.725 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-06T08:21:46.725 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-06T08:21:46.725 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:46.725 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-06T08:21:46.725 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-06T08:21:46.725 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:46.726 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-06T08:21:46.726 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:46.726 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-06T08:21:46.726 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-06T08:21:46.726 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-06T08:21:46.726 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-06T08:21:46.726 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-06T08:21:46.726 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-06T08:21:46.726 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:46.727 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-06T08:21:46.727 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-06T08:21:46.727 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-06T08:21:46.727 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-06T08:21:46.727 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-06T08:21:46.727 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-06T08:21:46.727 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-06T08:21:46.727 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-06T08:21:46.727 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-06T08:21:46.728 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:46.728 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-06T08:21:46.728 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:46.728 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-06T08:21:46.728 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-06T08:21:46.728 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:46.728 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-06T08:21:46.728 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-06T08:21:46.729 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-06T08:21:46.729 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-06T08:21:46.729 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-06T08:21:46.729 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-06T08:21:46.729 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "restful" 2024-06-06T08:21:46.729 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-06T08:21:46.729 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T08:21:46.729 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:46.730 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-06T08:21:46.730 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:46.730 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modified": "2024-06-06T08:21:26.883287+0000", 2024-06-06T08:21:46.730 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T08:21:46.730 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:46.730 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-06T08:21:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-06T08:21:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.121+0000 7feb2af71640 1 Processor -- start 2024-06-06T08:21:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.121+0000 7feb2af71640 1 -- start start 2024-06-06T08:21:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.121+0000 7feb2af71640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb24101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.121+0000 7feb2af71640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb241015a0 con 0x7feb24100c50 2024-06-06T08:21:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.121+0000 7feb28d0d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb24101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.122+0000 7feb28d0d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb24101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59324/0 (socket says 172.21.15.86:59324) 2024-06-06T08:21:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.122+0000 7feb28d0d640 1 -- 172.21.15.86:0/1243242381 learned_addr learned my addr 172.21.15.86:0/1243242381 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.122+0000 7feb28d0d640 1 -- 172.21.15.86:0/1243242381 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb241016e0 con 0x7feb24100c50 2024-06-06T08:21:46.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.123+0000 7feb28d0d640 1 --2- 172.21.15.86:0/1243242381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb24101060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb1800baf0 tx=0x7feb1800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7c5bda08f701a server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:46.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.123+0000 7feb237fe640 1 -- 172.21.15.86:0/1243242381 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb180327a0 con 0x7feb24100c50 2024-06-06T08:21:46.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.123+0000 7feb237fe640 1 -- 172.21.15.86:0/1243242381 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7feb18032d90 con 0x7feb24100c50 2024-06-06T08:21:46.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.123+0000 7feb237fe640 1 -- 172.21.15.86:0/1243242381 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb18031e70 con 0x7feb24100c50 2024-06-06T08:21:46.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.123+0000 7feb2af71640 1 -- 172.21.15.86:0/1243242381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 msgr2=0x7feb24101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:46.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.123+0000 7feb2af71640 1 --2- 172.21.15.86:0/1243242381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb24101060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb1800baf0 tx=0x7feb1800b460 comp rx=0 tx=0).stop 2024-06-06T08:21:46.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.123+0000 7feb2af71640 1 -- 172.21.15.86:0/1243242381 shutdown_connections 2024-06-06T08:21:46.732 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.123+0000 7feb2af71640 1 --2- 172.21.15.86:0/1243242381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb24101060 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.124+0000 7feb2af71640 1 -- 172.21.15.86:0/1243242381 >> 172.21.15.86:0/1243242381 conn(0x7feb240fc430 msgr2=0x7feb240fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.124+0000 7feb2af71640 1 -- 172.21.15.86:0/1243242381 shutdown_connections 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.124+0000 7feb2af71640 1 -- 172.21.15.86:0/1243242381 wait complete. 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.124+0000 7feb2af71640 1 Processor -- start 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.124+0000 7feb2af71640 1 -- start start 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.124+0000 7feb2af71640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb2410e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.124+0000 7feb2af71640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2410eec0 con 0x7feb24100c50 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb28d0d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb2410e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb28d0d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb2410e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59332/0 (socket says 172.21.15.86:59332) 2024-06-06T08:21:46.733 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb28d0d640 1 -- 172.21.15.86:0/1278202230 learned_addr learned my addr 172.21.15.86:0/1278202230 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb28d0d640 1 -- 172.21.15.86:0/1278202230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1800b700 con 0x7feb24100c50 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb28d0d640 1 --2- 172.21.15.86:0/1278202230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb2410e980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb18006f00 tx=0x7feb1800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb217fa640 1 -- 172.21.15.86:0/1278202230 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb18032a30 con 0x7feb24100c50 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb217fa640 1 -- 172.21.15.86:0/1278202230 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7feb18032b90 con 0x7feb24100c50 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb2af71640 1 -- 172.21.15.86:0/1278202230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2410f0c0 con 0x7feb24100c50 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb217fa640 1 -- 172.21.15.86:0/1278202230 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb1803a730 con 0x7feb24100c50 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.125+0000 7feb2af71640 1 -- 172.21.15.86:0/1278202230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2410f4d0 con 0x7feb24100c50 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.126+0000 7feb217fa640 1 -- 172.21.15.86:0/1278202230 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7feb1803abd0 con 0x7feb24100c50 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.126+0000 7feb217fa640 1 -- 172.21.15.86:0/1278202230 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7feb1804fe30 con 0x7feb24100c50 2024-06-06T08:21:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.126+0000 7feb2af71640 1 -- 172.21.15.86:0/1278202230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2404b480 con 0x7feb24100c50 2024-06-06T08:21:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.128+0000 7feb217fa640 1 -- 172.21.15.86:0/1278202230 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7feb18036020 con 0x7feb24100c50 2024-06-06T08:21:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.191+0000 7feb2af71640 1 -- 172.21.15.86:0/1278202230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7feb241029c0 con 0x7feb24100c50 2024-06-06T08:21:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.191+0000 7feb217fa640 1 -- 172.21.15.86:0/1278202230 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7feb1803bce0 con 0x7feb24100c50 2024-06-06T08:21:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.192+0000 7feb2af71640 1 -- 172.21.15.86:0/1278202230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 msgr2=0x7feb2410e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.192+0000 7feb2af71640 1 --2- 172.21.15.86:0/1278202230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb2410e980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb18006f00 tx=0x7feb1800b460 comp rx=0 tx=0).stop 2024-06-06T08:21:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.192+0000 7feb2af71640 1 -- 172.21.15.86:0/1278202230 shutdown_connections 2024-06-06T08:21:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.192+0000 7feb2af71640 1 --2- 172.21.15.86:0/1278202230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100c50 0x7feb2410e980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.192+0000 7feb2af71640 1 -- 172.21.15.86:0/1278202230 >> 172.21.15.86:0/1278202230 conn(0x7feb240fc430 msgr2=0x7feb24105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.192+0000 7feb2af71640 1 -- 172.21.15.86:0/1278202230 shutdown_connections 2024-06-06T08:21:46.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:46.192+0000 7feb2af71640 1 -- 172.21.15.86:0/1278202230 wait complete. 2024-06-06T08:21:46.736 INFO:teuthology.orchestra.run.smithi086.stdout:mgr not available, waiting (2/15)... 2024-06-06T08:21:47.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: Activating manager daemon smithi086.mfkgze 2024-06-06T08:21:47.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: mgrmap e2: smithi086.mfkgze(active, starting, since 0.00262385s) 2024-06-06T08:21:47.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T08:21:47.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T08:21:47.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T08:21:47.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-06-06T08:21:47.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr metadata", "who": "smithi086.mfkgze", "id": "smithi086.mfkgze"}]: dispatch 2024-06-06T08:21:47.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: Manager daemon smithi086.mfkgze is now available 2024-06-06T08:21:47.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.mfkgze/mirror_snapshot_schedule"}]: dispatch 2024-06-06T08:21:47.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.mfkgze/trash_purge_schedule"}]: dispatch 2024-06-06T08:21:47.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' 2024-06-06T08:21:47.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' 2024-06-06T08:21:47.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:47 smithi086 ceph-mon[29114]: from='mgr.14100 172.21.15.86:0/1953591764' entity='mgr.smithi086.mfkgze' 2024-06-06T08:21:49.098 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:49 smithi086 ceph-mon[29114]: mgrmap e3: smithi086.mfkgze(active, since 1.00644s) 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsid": "a8550dee-23dd-11ef-bc9b-c7b262605968", 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "health": { 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-06T08:21:49.892 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 0 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "smithi086" 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-06T08:21:49.893 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-06T08:21:49.901 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-06T08:21:49.901 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-06T08:21:49.901 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-06T08:21:49.901 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:49.902 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "restful" 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T08:21:49.903 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modified": "2024-06-06T08:21:26.883287+0000", 2024-06-06T08:21:49.904 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T08:21:49.904 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-06T08:21:49.904 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-06T08:21:49.904 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-06T08:21:49.904 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.098+0000 7f5567507640 1 Processor -- start 2024-06-06T08:21:49.904 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.098+0000 7f5567507640 1 -- start start 2024-06-06T08:21:49.904 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.098+0000 7f5567507640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f55601015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:49.904 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.098+0000 7f5567507640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560101b20 con 0x7f55601011d0 2024-06-06T08:21:49.904 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.099+0000 7f55652a3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f55601015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.099+0000 7f55652a3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f55601015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55156/0 (socket says 172.21.15.86:55156) 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.099+0000 7f55652a3640 1 -- 172.21.15.86:0/1413560645 learned_addr learned my addr 172.21.15.86:0/1413560645 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.099+0000 7f55652a3640 1 -- 172.21.15.86:0/1413560645 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5560101c60 con 0x7f55601011d0 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.100+0000 7f55652a3640 1 --2- 172.21.15.86:0/1413560645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f55601015e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5550009840 tx=0x7f55500091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5916c38c66def72 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.100+0000 7f5557fff640 1 -- 172.21.15.86:0/1413560645 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5550032440 con 0x7f55601011d0 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.100+0000 7f5557fff640 1 -- 172.21.15.86:0/1413560645 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5550032a30 con 0x7f55601011d0 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.100+0000 7f5557fff640 1 -- 172.21.15.86:0/1413560645 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5550031ad0 con 0x7f55601011d0 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.101+0000 7f5567507640 1 -- 172.21.15.86:0/1413560645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 msgr2=0x7f55601015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.101+0000 7f5567507640 1 --2- 172.21.15.86:0/1413560645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f55601015e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5550009840 tx=0x7f55500091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.101+0000 7f5567507640 1 -- 172.21.15.86:0/1413560645 shutdown_connections 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.101+0000 7f5567507640 1 --2- 172.21.15.86:0/1413560645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f55601015e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.101+0000 7f5567507640 1 -- 172.21.15.86:0/1413560645 >> 172.21.15.86:0/1413560645 conn(0x7f55600fbc20 msgr2=0x7f55600fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.101+0000 7f5567507640 1 -- 172.21.15.86:0/1413560645 shutdown_connections 2024-06-06T08:21:49.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.101+0000 7f5567507640 1 -- 172.21.15.86:0/1413560645 wait complete. 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f5567507640 1 Processor -- start 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f5567507640 1 -- start start 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f5567507640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f5560106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f5567507640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560107410 con 0x7f55601011d0 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f55652a3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f5560106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f55652a3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f5560106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55166/0 (socket says 172.21.15.86:55166) 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f55652a3640 1 -- 172.21.15.86:0/3338500908 learned_addr learned my addr 172.21.15.86:0/3338500908 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f55652a3640 1 -- 172.21.15.86:0/3338500908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5550009450 con 0x7f55601011d0 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f55652a3640 1 --2- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f5560106ed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f55500317b0 tx=0x7f5550008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f5555ffb640 1 -- 172.21.15.86:0/3338500908 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5550031e90 con 0x7f55601011d0 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f5555ffb640 1 -- 172.21.15.86:0/3338500908 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f555003daf0 con 0x7f55601011d0 2024-06-06T08:21:49.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.102+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55601054a0 con 0x7f55601011d0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.103+0000 7f5555ffb640 1 -- 172.21.15.86:0/3338500908 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f555003baa0 con 0x7f55601011d0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.103+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5560105910 con 0x7f55601011d0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.103+0000 7f5555ffb640 1 -- 172.21.15.86:0/3338500908 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f5550039070 con 0x7f55601011d0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.104+0000 7f5555ffb640 1 --2- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f554c0347c0 0x7f554c036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.104+0000 7f5555ffb640 1 -- 172.21.15.86:0/3338500908 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f55500675a0 con 0x7f55601011d0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.104+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556005ead0 con 0x7f55601011d0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.104+0000 7f5564aa2640 1 --2- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f554c0347c0 0x7f554c036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.104+0000 7f5564aa2640 1 --2- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f554c0347c0 0x7f554c036c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f555c006fd0 tx=0x7f555c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.109+0000 7f5555ffb640 1 -- 172.21.15.86:0/3338500908 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5550041080 con 0x7f55601011d0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.335+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f556010a820 con 0x7f55601011d0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.335+0000 7f5555ffb640 1 -- 172.21.15.86:0/3338500908 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f5550037070 con 0x7f55601011d0 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f554c0347c0 msgr2=0x7f554c036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 --2- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f554c0347c0 0x7f554c036c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f555c006fd0 tx=0x7f555c006db0 comp rx=0 tx=0).stop 2024-06-06T08:21:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 msgr2=0x7f5560106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:49.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 --2- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f5560106ed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f55500317b0 tx=0x7f5550008040 comp rx=0 tx=0).stop 2024-06-06T08:21:49.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 shutdown_connections 2024-06-06T08:21:49.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 --2- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f554c0347c0 0x7f554c036c70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:49.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 --2- 172.21.15.86:0/3338500908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55601011d0 0x7f5560106ed0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:49.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 >> 172.21.15.86:0/3338500908 conn(0x7f55600fbc20 msgr2=0x7f55600fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:49.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 shutdown_connections 2024-06-06T08:21:49.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:49.337+0000 7f5567507640 1 -- 172.21.15.86:0/3338500908 wait complete. 2024-06-06T08:21:49.908 INFO:teuthology.orchestra.run.smithi086.stdout:mgr is available 2024-06-06T08:21:50.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:50 smithi086 ceph-mon[29114]: mgrmap e4: smithi086.mfkgze(active, since 2s) 2024-06-06T08:21:50.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:50 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3338500908' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-06T08:21:51.069 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:51.069 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [global] 2024-06-06T08:21:51.069 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout fsid = a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:21:51.069 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-06T08:21:51.069 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-06T08:21:51.069 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-06T08:21:51.069 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-06T08:21:51.069 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:51.069 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [osd] 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.291+0000 7f13226e7640 1 Processor -- start 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.291+0000 7f13226e7640 1 -- start start 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.292+0000 7f13226e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.292+0000 7f13226e7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c1026e0 con 0x7f131c101d90 2024-06-06T08:21:51.070 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.292+0000 7f131bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.292+0000 7f131bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55174/0 (socket says 172.21.15.86:55174) 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.292+0000 7f131bfff640 1 -- 172.21.15.86:0/3103840110 learned_addr learned my addr 172.21.15.86:0/3103840110 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.292+0000 7f131bfff640 1 -- 172.21.15.86:0/3103840110 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c102820 con 0x7f131c101d90 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f131bfff640 1 --2- 172.21.15.86:0/3103840110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f130c009840 tx=0x7f130c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b8672190165740 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f131affd640 1 -- 172.21.15.86:0/3103840110 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f130c032440 con 0x7f131c101d90 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f131affd640 1 -- 172.21.15.86:0/3103840110 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f130c032a30 con 0x7f131c101d90 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f131affd640 1 -- 172.21.15.86:0/3103840110 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f130c031ad0 con 0x7f131c101d90 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f13226e7640 1 -- 172.21.15.86:0/3103840110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 msgr2=0x7f131c1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f13226e7640 1 --2- 172.21.15.86:0/3103840110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f130c009840 tx=0x7f130c0091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f13226e7640 1 -- 172.21.15.86:0/3103840110 shutdown_connections 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f13226e7640 1 --2- 172.21.15.86:0/3103840110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f13226e7640 1 -- 172.21.15.86:0/3103840110 >> 172.21.15.86:0/3103840110 conn(0x7f131c0fd5b0 msgr2=0x7f131c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:51.071 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.293+0000 7f13226e7640 1 -- 172.21.15.86:0/3103840110 shutdown_connections 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.294+0000 7f13226e7640 1 -- 172.21.15.86:0/3103840110 wait complete. 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.294+0000 7f13226e7640 1 Processor -- start 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.294+0000 7f13226e7640 1 -- start start 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.294+0000 7f13226e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c110e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.294+0000 7f13226e7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c1113d0 con 0x7f131c101d90 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f131bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c110e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f131bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c110e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55178/0 (socket says 172.21.15.86:55178) 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f131bfff640 1 -- 172.21.15.86:0/2683297211 learned_addr learned my addr 172.21.15.86:0/2683297211 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f131bfff640 1 -- 172.21.15.86:0/2683297211 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c009450 con 0x7f131c101d90 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f131bfff640 1 --2- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c110e90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f130c0317b0 tx=0x7f130c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:51.072 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f1318ff9640 1 -- 172.21.15.86:0/2683297211 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f130c031e90 con 0x7f131c101d90 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f1318ff9640 1 -- 172.21.15.86:0/2683297211 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f130c03daf0 con 0x7f131c101d90 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c1115d0 con 0x7f131c101d90 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f1318ff9640 1 -- 172.21.15.86:0/2683297211 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f130c03baa0 con 0x7f131c101d90 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.295+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c111fd0 con 0x7f131c101d90 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.296+0000 7f1318ff9640 1 -- 172.21.15.86:0/2683297211 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f130c039070 con 0x7f131c101d90 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.296+0000 7f1318ff9640 1 --2- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f1308034810 0x7f1308036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.296+0000 7f1318ff9640 1 -- 172.21.15.86:0/2683297211 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f130c0674b0 con 0x7f131c101d90 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.296+0000 7f131b7fe640 1 --2- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f1308034810 0x7f1308036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.296+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c05ead0 con 0x7f131c101d90 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.297+0000 7f131b7fe640 1 --2- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f1308034810 0x7f1308036cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1304006fd0 tx=0x7f1304006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.301+0000 7f1318ff9640 1 -- 172.21.15.86:0/2683297211 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f130c041080 con 0x7f131c101d90 2024-06-06T08:21:51.073 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.486+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f131c063ed0 con 0x7f131c101d90 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.487+0000 7f1318ff9640 1 -- 172.21.15.86:0/2683297211 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f130c066190 con 0x7f131c101d90 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f1308034810 msgr2=0x7f1308036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 --2- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f1308034810 0x7f1308036cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1304006fd0 tx=0x7f1304006db0 comp rx=0 tx=0).stop 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 msgr2=0x7f131c110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 --2- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c110e90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f130c0317b0 tx=0x7f130c008040 comp rx=0 tx=0).stop 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 shutdown_connections 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 --2- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f1308034810 0x7f1308036cc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 --2- 172.21.15.86:0/2683297211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c101d90 0x7f131c110e90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 >> 172.21.15.86:0/2683297211 conn(0x7f131c0fd5b0 msgr2=0x7f131c0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 shutdown_connections 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:50.489+0000 7f13226e7640 1 -- 172.21.15.86:0/2683297211 wait complete. 2024-06-06T08:21:51.074 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling cephadm module... 2024-06-06T08:21:51.189 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:51 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2683297211' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-06T08:21:52.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:52 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/734164240' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-06T08:21:52.565 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.431+0000 7f86dbddb640 1 Processor -- start 2024-06-06T08:21:52.565 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.432+0000 7f86dbddb640 1 -- start start 2024-06-06T08:21:52.565 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.432+0000 7f86dbddb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4101670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:52.565 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.432+0000 7f86dbddb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4101bb0 con 0x7f86d4101260 2024-06-06T08:21:52.565 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.432+0000 7f86d9b77640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4101670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.432+0000 7f86d9b77640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4101670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55190/0 (socket says 172.21.15.86:55190) 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.432+0000 7f86d9b77640 1 -- 172.21.15.86:0/1759685894 learned_addr learned my addr 172.21.15.86:0/1759685894 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.432+0000 7f86d9b77640 1 -- 172.21.15.86:0/1759685894 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d4101cf0 con 0x7f86d4101260 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.433+0000 7f86d9b77640 1 --2- 172.21.15.86:0/1759685894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4101670 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009840 tx=0x7f86d00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2bced96f8572b2 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.433+0000 7f86d8b75640 1 -- 172.21.15.86:0/1759685894 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d0032440 con 0x7f86d4101260 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.433+0000 7f86d8b75640 1 -- 172.21.15.86:0/1759685894 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f86d0032a30 con 0x7f86d4101260 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.433+0000 7f86d8b75640 1 -- 172.21.15.86:0/1759685894 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d0031ad0 con 0x7f86d4101260 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 -- 172.21.15.86:0/1759685894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 msgr2=0x7f86d4101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 --2- 172.21.15.86:0/1759685894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4101670 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009840 tx=0x7f86d00091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 -- 172.21.15.86:0/1759685894 shutdown_connections 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 --2- 172.21.15.86:0/1759685894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4101670 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 -- 172.21.15.86:0/1759685894 >> 172.21.15.86:0/1759685894 conn(0x7f86d40fbbf0 msgr2=0x7f86d40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 -- 172.21.15.86:0/1759685894 shutdown_connections 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 -- 172.21.15.86:0/1759685894 wait complete. 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 Processor -- start 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 -- start start 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4106e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86dbddb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d41073c0 con 0x7f86d4101260 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86d9b77640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86d9b77640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55200/0 (socket says 172.21.15.86:55200) 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.434+0000 7f86d9b77640 1 -- 172.21.15.86:0/734164240 learned_addr learned my addr 172.21.15.86:0/734164240 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:52.567 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.435+0000 7f86d9b77640 1 -- 172.21.15.86:0/734164240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0009450 con 0x7f86d4101260 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.435+0000 7f86d9b77640 1 --2- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4106e80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f86d00317b0 tx=0x7f86d0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.435+0000 7f86ca7fc640 1 -- 172.21.15.86:0/734164240 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d0031e90 con 0x7f86d4101260 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.435+0000 7f86ca7fc640 1 -- 172.21.15.86:0/734164240 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f86d003daf0 con 0x7f86d4101260 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.435+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d41054a0 con 0x7f86d4101260 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.435+0000 7f86ca7fc640 1 -- 172.21.15.86:0/734164240 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d003baa0 con 0x7f86d4101260 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.435+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d4105910 con 0x7f86d4101260 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.436+0000 7f86ca7fc640 1 -- 172.21.15.86:0/734164240 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f86d0039070 con 0x7f86d4101260 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.436+0000 7f86ca7fc640 1 --2- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f86c00347c0 0x7f86c0036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.436+0000 7f86ca7fc640 1 -- 172.21.15.86:0/734164240 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f86d0067880 con 0x7f86d4101260 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.436+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d405ead0 con 0x7f86d4101260 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.436+0000 7f86d9376640 1 --2- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f86c00347c0 0x7f86c0036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.437+0000 7f86d9376640 1 --2- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f86c00347c0 0x7f86c0036c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f86c4006fd0 tx=0x7f86c4006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:52.568 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.441+0000 7f86ca7fc640 1 -- 172.21.15.86:0/734164240 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f86d0041020 con 0x7f86d4101260 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:51.650+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f86d404b480 con 0x7f86d4101260 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.029+0000 7f86ca7fc640 1 -- 172.21.15.86:0/734164240 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f86d003dcd0 con 0x7f86d4101260 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.029+0000 7f86ca7fc640 1 -- 172.21.15.86:0/734164240 <== mon.0 v2:172.21.15.86:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f86d0044df0 con 0x7f86d4101260 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f86c00347c0 msgr2=0x7f86c0036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 --2- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f86c00347c0 0x7f86c0036c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f86c4006fd0 tx=0x7f86c4006db0 comp rx=0 tx=0).stop 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 msgr2=0x7f86d4106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 --2- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4106e80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f86d00317b0 tx=0x7f86d0008040 comp rx=0 tx=0).stop 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 shutdown_connections 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 --2- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7f86c00347c0 0x7f86c0036c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 --2- 172.21.15.86:0/734164240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d4101260 0x7f86d4106e80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 >> 172.21.15.86:0/734164240 conn(0x7f86d40fbbf0 msgr2=0x7f86d40fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:52.569 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 shutdown_connections 2024-06-06T08:21:52.570 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.032+0000 7f86dbddb640 1 -- 172.21.15.86:0/734164240 wait complete. 2024-06-06T08:21:53.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:53 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/734164240' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-06T08:21:53.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:53 smithi086 ceph-mon[29114]: mgrmap e5: smithi086.mfkgze(active, since 5s) 2024-06-06T08:21:53.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-06T08:21:53.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-06T08:21:53.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-06T08:21:53.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "active_name": "smithi086.mfkgze", 2024-06-06T08:21:53.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-06T08:21:53.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.986+0000 7fe5c45df640 1 Processor -- start 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.986+0000 7fe5c45df640 1 -- start start 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.987+0000 7fe5c45df640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.987+0000 7fe5c45df640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc075f70 con 0x7fe5bc075b60 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.987+0000 7fe5c237b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.987+0000 7fe5c237b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55236/0 (socket says 172.21.15.86:55236) 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.987+0000 7fe5c237b640 1 -- 172.21.15.86:0/930224491 learned_addr learned my addr 172.21.15.86:0/930224491 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.987+0000 7fe5c237b640 1 -- 172.21.15.86:0/930224491 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc104ec0 con 0x7fe5bc075b60 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.988+0000 7fe5c237b640 1 --2- 172.21.15.86:0/930224491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc104980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009840 tx=0x7fe5b80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9febb99bda5268e server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.988+0000 7fe5c1379640 1 -- 172.21.15.86:0/930224491 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5b8032440 con 0x7fe5bc075b60 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.988+0000 7fe5c1379640 1 -- 172.21.15.86:0/930224491 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe5b8032a30 con 0x7fe5bc075b60 2024-06-06T08:21:53.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.988+0000 7fe5c1379640 1 -- 172.21.15.86:0/930224491 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5b8031ad0 con 0x7fe5bc075b60 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.989+0000 7fe5c45df640 1 -- 172.21.15.86:0/930224491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 msgr2=0x7fe5bc104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.989+0000 7fe5c45df640 1 --2- 172.21.15.86:0/930224491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc104980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009840 tx=0x7fe5b80091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.989+0000 7fe5c45df640 1 -- 172.21.15.86:0/930224491 shutdown_connections 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.989+0000 7fe5c45df640 1 --2- 172.21.15.86:0/930224491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc104980 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.989+0000 7fe5c45df640 1 -- 172.21.15.86:0/930224491 >> 172.21.15.86:0/930224491 conn(0x7fe5bc0fc430 msgr2=0x7fe5bc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.989+0000 7fe5c45df640 1 -- 172.21.15.86:0/930224491 shutdown_connections 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.989+0000 7fe5c45df640 1 -- 172.21.15.86:0/930224491 wait complete. 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.989+0000 7fe5c45df640 1 Processor -- start 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.989+0000 7fe5c45df640 1 -- start start 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5c45df640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc110b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5c45df640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc111040 con 0x7fe5bc075b60 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5c237b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc110b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5c237b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc110b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55246/0 (socket says 172.21.15.86:55246) 2024-06-06T08:21:53.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5c237b640 1 -- 172.21.15.86:0/3559005929 learned_addr learned my addr 172.21.15.86:0/3559005929 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5c237b640 1 -- 172.21.15.86:0/3559005929 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b8009450 con 0x7fe5bc075b60 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5c237b640 1 --2- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc110b00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8006f00 tx=0x7fe5b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5b2ffd640 1 -- 172.21.15.86:0/3559005929 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5b803d450 con 0x7fe5bc075b60 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5b2ffd640 1 -- 172.21.15.86:0/3559005929 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe5b803da40 con 0x7fe5bc075b60 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5bc111240 con 0x7fe5bc075b60 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5b2ffd640 1 -- 172.21.15.86:0/3559005929 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5b803bb10 con 0x7fe5bc075b60 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.990+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5bc1116b0 con 0x7fe5bc075b60 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.991+0000 7fe5b2ffd640 1 -- 172.21.15.86:0/3559005929 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fe5b8039070 con 0x7fe5bc075b60 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.991+0000 7fe5b2ffd640 1 --2- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fe5a8034860 0x7fe5a8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.991+0000 7fe5b2ffd640 1 -- 172.21.15.86:0/3559005929 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe5b8067240 con 0x7fe5bc075b60 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.991+0000 7fe5c1b7a640 1 -- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fe5a8034860 msgr2=0x7fe5a8036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/3222204828 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.991+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5bc05ead0 con 0x7fe5bc075b60 2024-06-06T08:21:53.833 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.991+0000 7fe5c1b7a640 1 --2- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fe5a8034860 0x7fe5a8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:52.996+0000 7fe5b2ffd640 1 -- 172.21.15.86:0/3559005929 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe5b8041080 con 0x7fe5bc075b60 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.192+0000 7fe5c1b7a640 1 -- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fe5a8034860 msgr2=0x7fe5a8036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/3222204828 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.192+0000 7fe5c1b7a640 1 --2- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fe5a8034860 0x7fe5a8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.211+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe5bc111e00 con 0x7fe5bc075b60 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.212+0000 7fe5b2ffd640 1 -- 172.21.15.86:0/3559005929 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fe5b8037020 con 0x7fe5bc075b60 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fe5a8034860 msgr2=0x7fe5a8036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 --2- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fe5a8034860 0x7fe5a8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 msgr2=0x7fe5bc110b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 --2- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc110b00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8006f00 tx=0x7fe5b8008040 comp rx=0 tx=0).stop 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 shutdown_connections 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 --2- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fe5a8034860 0x7fe5a8036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 --2- 172.21.15.86:0/3559005929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5bc075b60 0x7fe5bc110b00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 >> 172.21.15.86:0/3559005929 conn(0x7fe5bc0fc430 msgr2=0x7fe5bc0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:53.834 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 shutdown_connections 2024-06-06T08:21:53.835 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:53.214+0000 7fe5c45df640 1 -- 172.21.15.86:0/3559005929 wait complete. 2024-06-06T08:21:53.835 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for the mgr to restart... 2024-06-06T08:21:53.835 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr epoch 5... 2024-06-06T08:21:54.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:54 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3559005929' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-06T08:21:56.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: Active manager daemon smithi086.mfkgze restarted 2024-06-06T08:21:56.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: Activating manager daemon smithi086.mfkgze 2024-06-06T08:21:56.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: osdmap e2: 0 total, 0 up, 0 in 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: mgrmap e6: smithi086.mfkgze(active, starting, since 0.00314032s) 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr metadata", "who": "smithi086.mfkgze", "id": "smithi086.mfkgze"}]: dispatch 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: Manager daemon smithi086.mfkgze is now available 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:21:56.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.mfkgze/mirror_snapshot_schedule"}]: dispatch 2024-06-06T08:21:56.837 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:56 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.mfkgze/trash_purge_schedule"}]: dispatch 2024-06-06T08:21:57.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:57 smithi086 ceph-mon[29114]: Found migration_current of "None". Setting to last migration. 2024-06-06T08:21:57.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:57 smithi086 ceph-mon[29114]: mgrmap e7: smithi086.mfkgze(active, since 1.00517s) 2024-06-06T08:21:58.245 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-06T08:21:58.245 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-06T08:21:58.245 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-06T08:21:58.245 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-06T08:21:58.245 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.226+0000 7fcbe7ed6640 1 Processor -- start 2024-06-06T08:21:58.245 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.226+0000 7fcbe7ed6640 1 -- start start 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.226+0000 7fcbe7ed6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.226+0000 7fcbe7ed6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe0075f70 con 0x7fcbe0075b60 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.226+0000 7fcbe5c72640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.226+0000 7fcbe5c72640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55258/0 (socket says 172.21.15.86:55258) 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.226+0000 7fcbe5c72640 1 -- 172.21.15.86:0/2679864065 learned_addr learned my addr 172.21.15.86:0/2679864065 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.226+0000 7fcbe5c72640 1 -- 172.21.15.86:0/2679864065 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe0104ec0 con 0x7fcbe0075b60 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.227+0000 7fcbe5c72640 1 --2- 172.21.15.86:0/2679864065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0104980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009840 tx=0x7fcbdc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0033d1e6939bc96 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.227+0000 7fcbe4c70640 1 -- 172.21.15.86:0/2679864065 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbdc032440 con 0x7fcbe0075b60 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.227+0000 7fcbe4c70640 1 -- 172.21.15.86:0/2679864065 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcbdc032a30 con 0x7fcbe0075b60 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.227+0000 7fcbe4c70640 1 -- 172.21.15.86:0/2679864065 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbdc031ad0 con 0x7fcbe0075b60 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.228+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2679864065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 msgr2=0x7fcbe0104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:58.246 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.228+0000 7fcbe7ed6640 1 --2- 172.21.15.86:0/2679864065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0104980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009840 tx=0x7fcbdc0091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.228+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2679864065 shutdown_connections 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.228+0000 7fcbe7ed6640 1 --2- 172.21.15.86:0/2679864065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0104980 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.228+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2679864065 >> 172.21.15.86:0/2679864065 conn(0x7fcbe00fc430 msgr2=0x7fcbe00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.228+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2679864065 shutdown_connections 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.228+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2679864065 wait complete. 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.229+0000 7fcbe7ed6640 1 Processor -- start 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.229+0000 7fcbe7ed6640 1 -- start start 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.229+0000 7fcbe7ed6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0110b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.229+0000 7fcbe7ed6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe0111060 con 0x7fcbe0075b60 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.229+0000 7fcbe5c72640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0110b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.229+0000 7fcbe5c72640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0110b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55270/0 (socket says 172.21.15.86:55270) 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.229+0000 7fcbe5c72640 1 -- 172.21.15.86:0/2016780567 learned_addr learned my addr 172.21.15.86:0/2016780567 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.229+0000 7fcbe5c72640 1 -- 172.21.15.86:0/2016780567 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc009450 con 0x7fcbe0075b60 2024-06-06T08:21:58.247 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.229+0000 7fcbe5c72640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0110b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc031b90 tx=0x7fcbdc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.230+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbdc0328b0 con 0x7fcbe0075b60 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.230+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcbdc03d8e0 con 0x7fcbe0075b60 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.230+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbdc03ba20 con 0x7fcbe0075b60 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.230+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe0111260 con 0x7fcbe0075b60 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.230+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe01116d0 con 0x7fcbe0075b60 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.231+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fcbdc039070 con 0x7fcbe0075b60 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.231+0000 7fcbd67fc640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 0x7fcbcc036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.231+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 --> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbe005ead0 con 0x7fcbcc034810 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.231+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fcbdc067550 con 0x7fcbe0075b60 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.231+0000 7fcbe5471640 1 -- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 msgr2=0x7fcbcc036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/3222204828 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.231+0000 7fcbe5471640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 0x7fcbcc036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T08:21:58.248 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.432+0000 7fcbe5471640 1 -- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 msgr2=0x7fcbcc036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/3222204828 2024-06-06T08:21:58.249 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.432+0000 7fcbe5471640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 0x7fcbcc036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T08:21:58.249 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.833+0000 7fcbe5471640 1 -- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 msgr2=0x7fcbcc036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/3222204828 2024-06-06T08:21:58.249 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:54.833+0000 7fcbe5471640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 0x7fcbcc036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-06T08:21:58.250 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:55.634+0000 7fcbe5471640 1 -- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 msgr2=0x7fcbcc036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/3222204828 2024-06-06T08:21:58.250 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:55.634+0000 7fcbe5471640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 0x7fcbcc036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-06T08:21:58.250 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:56.546+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mgrmap(e 6) v1 ==== 41713+0+0 (secure 0 0 0) 0x7fcbdc03bc90 con 0x7fcbe0075b60 2024-06-06T08:21:58.251 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:56.546+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 msgr2=0x7fcbcc036cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T08:21:58.251 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:56.546+0000 7fcbd67fc640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 0x7fcbcc036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:58.251 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.547+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7fcbdc046590 con 0x7fcbe0075b60 2024-06-06T08:21:58.251 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.547+0000 7fcbd67fc640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fcbcc037fd0 0x7fcbcc03a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:58.251 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.547+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbe005ead0 con 0x7fcbcc037fd0 2024-06-06T08:21:58.251 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.548+0000 7fcbe5471640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fcbcc037fd0 0x7fcbcc03a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:58.252 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.548+0000 7fcbe5471640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fcbcc037fd0 0x7fcbcc03a3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0003940 tx=0x7fcbd0009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:58.252 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.549+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fcbe005ead0 con 0x7fcbcc037fd0 2024-06-06T08:21:58.252 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.554+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcbe004b480 con 0x7fcbcc037fd0 2024-06-06T08:21:58.252 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.554+0000 7fcbd67fc640 1 -- 172.21.15.86:0/2016780567 <== mgr.14118 v2:172.21.15.86:6800/933334455 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcbe004b480 con 0x7fcbcc037fd0 2024-06-06T08:21:58.252 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fcbcc037fd0 msgr2=0x7fcbcc03a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:58.252 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fcbcc037fd0 0x7fcbcc03a3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0003940 tx=0x7fcbd0009350 comp rx=0 tx=0).stop 2024-06-06T08:21:58.253 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 msgr2=0x7fcbe0110b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:58.253 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0110b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc031b90 tx=0x7fcbdc008040 comp rx=0 tx=0).stop 2024-06-06T08:21:58.253 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 shutdown_connections 2024-06-06T08:21:58.254 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fcbcc037fd0 0x7fcbcc03a3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:58.254 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:6800/3222204828,v1:172.21.15.86:6801/3222204828] conn(0x7fcbcc034810 0x7fcbcc036cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:58.254 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 --2- 172.21.15.86:0/2016780567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe0075b60 0x7fcbe0110b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:58.254 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 >> 172.21.15.86:0/2016780567 conn(0x7fcbe00fc430 msgr2=0x7fcbe00fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:58.255 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 shutdown_connections 2024-06-06T08:21:58.255 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:57.555+0000 7fcbe7ed6640 1 -- 172.21.15.86:0/2016780567 wait complete. 2024-06-06T08:21:58.255 INFO:teuthology.orchestra.run.smithi086.stdout:mgr epoch 5 is available 2024-06-06T08:21:58.256 INFO:teuthology.orchestra.run.smithi086.stdout:Setting orchestrator backend to cephadm... 2024-06-06T08:21:58.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:58 smithi086 ceph-mon[29114]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-06T08:21:58.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:58 smithi086 ceph-mon[29114]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-06T08:21:58.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:58 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:21:58.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:58 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:21:59.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.615+0000 7fb7d8937640 1 Processor -- start 2024-06-06T08:21:59.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.615+0000 7fb7d8937640 1 -- start start 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.616+0000 7fb7d8937640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d41020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.616+0000 7fb7d8937640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4102600 con 0x7fb7d4101cb0 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.616+0000 7fb7d259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d41020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.616+0000 7fb7d259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d41020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43726/0 (socket says 172.21.15.86:43726) 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.616+0000 7fb7d259c640 1 -- 172.21.15.86:0/1594299412 learned_addr learned my addr 172.21.15.86:0/1594299412 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.616+0000 7fb7d259c640 1 -- 172.21.15.86:0/1594299412 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4102740 con 0x7fb7d4101cb0 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.617+0000 7fb7d259c640 1 --2- 172.21.15.86:0/1594299412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d41020c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009840 tx=0x7fb7c40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e24af5d5c59e6e0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.617+0000 7fb7d1d9b640 1 -- 172.21.15.86:0/1594299412 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c4032440 con 0x7fb7d4101cb0 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.617+0000 7fb7d1d9b640 1 -- 172.21.15.86:0/1594299412 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb7c4032a30 con 0x7fb7d4101cb0 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.617+0000 7fb7d1d9b640 1 -- 172.21.15.86:0/1594299412 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c4031ad0 con 0x7fb7d4101cb0 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.617+0000 7fb7d8937640 1 -- 172.21.15.86:0/1594299412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 msgr2=0x7fb7d41020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.617+0000 7fb7d8937640 1 --2- 172.21.15.86:0/1594299412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d41020c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009840 tx=0x7fb7c40091d0 comp rx=0 tx=0).stop 2024-06-06T08:21:59.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.617+0000 7fb7d8937640 1 -- 172.21.15.86:0/1594299412 shutdown_connections 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.617+0000 7fb7d8937640 1 --2- 172.21.15.86:0/1594299412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d41020c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.617+0000 7fb7d8937640 1 -- 172.21.15.86:0/1594299412 >> 172.21.15.86:0/1594299412 conn(0x7fb7d40fd490 msgr2=0x7fb7d40ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d8937640 1 -- 172.21.15.86:0/1594299412 shutdown_connections 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d8937640 1 -- 172.21.15.86:0/1594299412 wait complete. 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d8937640 1 Processor -- start 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d8937640 1 -- start start 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d8937640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d4110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d8937640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d41113e0 con 0x7fb7d4101cb0 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d4110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d4110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43730/0 (socket says 172.21.15.86:43730) 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d259c640 1 -- 172.21.15.86:0/3463453161 learned_addr learned my addr 172.21.15.86:0/3463453161 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:21:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.618+0000 7fb7d259c640 1 -- 172.21.15.86:0/3463453161 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4009450 con 0x7fb7d4101cb0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.619+0000 7fb7d259c640 1 --2- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d4110ea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40317b0 tx=0x7fb7c4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.619+0000 7fb7caffd640 1 -- 172.21.15.86:0/3463453161 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c4031e90 con 0x7fb7d4101cb0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.619+0000 7fb7caffd640 1 -- 172.21.15.86:0/3463453161 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb7c403daf0 con 0x7fb7d4101cb0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.619+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d41115e0 con 0x7fb7d4101cb0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.619+0000 7fb7caffd640 1 -- 172.21.15.86:0/3463453161 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c403baa0 con 0x7fb7d4101cb0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.619+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d4111fe0 con 0x7fb7d4101cb0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.619+0000 7fb7caffd640 1 -- 172.21.15.86:0/3463453161 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7fb7c4039070 con 0x7fb7d4101cb0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.620+0000 7fb7caffd640 1 --2- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7c0034730 0x7fb7c0036be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.620+0000 7fb7caffd640 1 -- 172.21.15.86:0/3463453161 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb7c40677f0 con 0x7fb7d4101cb0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.620+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d405ead0 con 0x7fb7d4101cb0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.620+0000 7fb7cbfff640 1 --2- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7c0034730 0x7fb7c0036be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.620+0000 7fb7cbfff640 1 --2- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7c0034730 0x7fb7c0036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc006fd0 tx=0x7fb7bc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:21:59.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.624+0000 7fb7caffd640 1 -- 172.21.15.86:0/3463453161 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb7c4041080 con 0x7fb7d4101cb0 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.798+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7d4103c20 con 0x7fb7c0034730 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.806+0000 7fb7caffd640 1 -- 172.21.15.86:0/3463453161 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb7d4103c20 con 0x7fb7c0034730 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7c0034730 msgr2=0x7fb7c0036be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 --2- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7c0034730 0x7fb7c0036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc006fd0 tx=0x7fb7bc006db0 comp rx=0 tx=0).stop 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 msgr2=0x7fb7d4110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 --2- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d4110ea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40317b0 tx=0x7fb7c4008040 comp rx=0 tx=0).stop 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 shutdown_connections 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 --2- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7c0034730 0x7fb7c0036be0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 --2- 172.21.15.86:0/3463453161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7d4101cb0 0x7fb7d4110ea0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 >> 172.21.15.86:0/3463453161 conn(0x7fb7d40fd490 msgr2=0x7fb7d40fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 shutdown_connections 2024-06-06T08:21:59.421 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:58.808+0000 7fb7d8937640 1 -- 172.21.15.86:0/3463453161 wait complete. 2024-06-06T08:21:59.966 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:59 smithi086 ceph-mon[29114]: [06/Jun/2024:08:21:58] ENGINE Bus STARTING 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:59 smithi086 ceph-mon[29114]: [06/Jun/2024:08:21:58] ENGINE Serving on https://172.21.15.86:7150 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:59 smithi086 ceph-mon[29114]: [06/Jun/2024:08:21:58] ENGINE Bus STARTED 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:59 smithi086 ceph-mon[29114]: [06/Jun/2024:08:21:58] ENGINE Error in HTTPServer.serve 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Traceback (most recent call last): 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self._run(expiration_interval) 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-06T08:21:59.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: s = self.context.wrap_socket( 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: return self.sslsocket_class._create( 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self.do_handshake() 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self._sslobj.do_handshake() 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:59 smithi086 ceph-mon[29114]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:59 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:59 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:21:59.968 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:21:59 smithi086 ceph-mon[29114]: mgrmap e8: smithi086.mfkgze(active, since 2s) 2024-06-06T08:22:00.540 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-06T08:22:00.540 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.790+0000 7fb80539a640 1 Processor -- start 2024-06-06T08:22:00.540 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.790+0000 7fb80539a640 1 -- start start 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.790+0000 7fb80539a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800104a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.790+0000 7fb80539a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb800075e80 con 0x7fb800075a70 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.790+0000 7fb7feffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.790+0000 7fb7feffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800104a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43744/0 (socket says 172.21.15.86:43744) 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.790+0000 7fb7feffd640 1 -- 172.21.15.86:0/2119319807 learned_addr learned my addr 172.21.15.86:0/2119319807 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.790+0000 7fb7feffd640 1 -- 172.21.15.86:0/2119319807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800104fc0 con 0x7fb800075a70 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.791+0000 7fb7feffd640 1 --2- 172.21.15.86:0/2119319807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800104a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009840 tx=0x7fb7f00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75cbd3255a305e8 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.791+0000 7fb7fe7fc640 1 -- 172.21.15.86:0/2119319807 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f0032440 con 0x7fb800075a70 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.791+0000 7fb7fe7fc640 1 -- 172.21.15.86:0/2119319807 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb7f0032a30 con 0x7fb800075a70 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.791+0000 7fb7fe7fc640 1 -- 172.21.15.86:0/2119319807 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f0031ad0 con 0x7fb800075a70 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.792+0000 7fb80539a640 1 -- 172.21.15.86:0/2119319807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 msgr2=0x7fb800104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.792+0000 7fb80539a640 1 --2- 172.21.15.86:0/2119319807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800104a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009840 tx=0x7fb7f00091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.792+0000 7fb80539a640 1 -- 172.21.15.86:0/2119319807 shutdown_connections 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.792+0000 7fb80539a640 1 --2- 172.21.15.86:0/2119319807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800104a80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.792+0000 7fb80539a640 1 -- 172.21.15.86:0/2119319807 >> 172.21.15.86:0/2119319807 conn(0x7fb8000fc490 msgr2=0x7fb8000fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.792+0000 7fb80539a640 1 -- 172.21.15.86:0/2119319807 shutdown_connections 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.792+0000 7fb80539a640 1 -- 172.21.15.86:0/2119319807 wait complete. 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.792+0000 7fb80539a640 1 Processor -- start 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.792+0000 7fb80539a640 1 -- start start 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb80539a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800110af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb80539a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb800111030 con 0x7fb800075a70 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb7feffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800110af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb7feffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800110af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43752/0 (socket says 172.21.15.86:43752) 2024-06-06T08:22:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb7feffd640 1 -- 172.21.15.86:0/3375153303 learned_addr learned my addr 172.21.15.86:0/3375153303 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb7feffd640 1 -- 172.21.15.86:0/3375153303 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f0009450 con 0x7fb800075a70 2024-06-06T08:22:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb7feffd640 1 --2- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800110af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0031b90 tx=0x7fb7f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb7f77fe640 1 -- 172.21.15.86:0/3375153303 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f00328b0 con 0x7fb800075a70 2024-06-06T08:22:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb7f77fe640 1 -- 172.21.15.86:0/3375153303 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb7f003d8e0 con 0x7fb800075a70 2024-06-06T08:22:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb800111230 con 0x7fb800075a70 2024-06-06T08:22:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb7f77fe640 1 -- 172.21.15.86:0/3375153303 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f003ba20 con 0x7fb800075a70 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.793+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8001116a0 con 0x7fb800075a70 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.794+0000 7fb7f77fe640 1 -- 172.21.15.86:0/3375153303 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fb7f0039070 con 0x7fb800075a70 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.794+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80004b480 con 0x7fb800075a70 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.794+0000 7fb7f77fe640 1 --2- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7ec034860 0x7fb7ec036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.794+0000 7fb7f77fe640 1 -- 172.21.15.86:0/3375153303 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb7f0067660 con 0x7fb800075a70 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.795+0000 7fb7f7fff640 1 --2- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7ec034860 0x7fb7ec036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.795+0000 7fb7f7fff640 1 --2- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7ec034860 0x7fb7ec036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8006fd0 tx=0x7fb7e8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.799+0000 7fb7f77fe640 1 -- 172.21.15.86:0/3375153303 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb7f003c590 con 0x7fb800075a70 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.967+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8001116a0 con 0x7fb7ec034860 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.968+0000 7fb7f77fe640 1 -- 172.21.15.86:0/3375153303 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb8001116a0 con 0x7fb7ec034860 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7ec034860 msgr2=0x7fb7ec036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 --2- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7ec034860 0x7fb7ec036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8006fd0 tx=0x7fb7e8006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 msgr2=0x7fb800110af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 --2- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800110af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0031b90 tx=0x7fb7f0008040 comp rx=0 tx=0).stop 2024-06-06T08:22:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 shutdown_connections 2024-06-06T08:22:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 --2- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb7ec034860 0x7fb7ec036d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 --2- 172.21.15.86:0/3375153303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb800075a70 0x7fb800110af0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 >> 172.21.15.86:0/3375153303 conn(0x7fb8000fc490 msgr2=0x7fb8000fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 shutdown_connections 2024-06-06T08:22:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:21:59.970+0000 7fb80539a640 1 -- 172.21.15.86:0/3375153303 wait complete. 2024-06-06T08:22:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:Generating ssh key... 2024-06-06T08:22:01.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:00 smithi086 ceph-mon[29114]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.920+0000 7fa80eeba640 1 Processor -- start 2024-06-06T08:22:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.920+0000 7fa80eeba640 1 -- start start 2024-06-06T08:22:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.921+0000 7fa80eeba640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa8081020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.921+0000 7fa80eeba640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa808102600 con 0x7fa808101cb0 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.921+0000 7fa80cc56640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa8081020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.921+0000 7fa80cc56640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa8081020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43758/0 (socket says 172.21.15.86:43758) 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.921+0000 7fa80cc56640 1 -- 172.21.15.86:0/755737899 learned_addr learned my addr 172.21.15.86:0/755737899 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.921+0000 7fa80cc56640 1 -- 172.21.15.86:0/755737899 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa808102740 con 0x7fa808101cb0 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.922+0000 7fa80cc56640 1 --2- 172.21.15.86:0/755737899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa8081020c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc009840 tx=0x7fa7fc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447f868c8eb816f9 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.922+0000 7fa8077fe640 1 -- 172.21.15.86:0/755737899 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7fc032440 con 0x7fa808101cb0 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.922+0000 7fa8077fe640 1 -- 172.21.15.86:0/755737899 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa7fc032a30 con 0x7fa808101cb0 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.922+0000 7fa8077fe640 1 -- 172.21.15.86:0/755737899 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7fc031ad0 con 0x7fa808101cb0 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.922+0000 7fa80eeba640 1 -- 172.21.15.86:0/755737899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 msgr2=0x7fa8081020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.922+0000 7fa80eeba640 1 --2- 172.21.15.86:0/755737899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa8081020c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc009840 tx=0x7fa7fc0091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.922+0000 7fa80eeba640 1 -- 172.21.15.86:0/755737899 shutdown_connections 2024-06-06T08:22:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.922+0000 7fa80eeba640 1 --2- 172.21.15.86:0/755737899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa8081020c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.922+0000 7fa80eeba640 1 -- 172.21.15.86:0/755737899 >> 172.21.15.86:0/755737899 conn(0x7fa8080fd4f0 msgr2=0x7fa8080ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.923+0000 7fa80eeba640 1 -- 172.21.15.86:0/755737899 shutdown_connections 2024-06-06T08:22:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.923+0000 7fa80eeba640 1 -- 172.21.15.86:0/755737899 wait complete. 2024-06-06T08:22:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.923+0000 7fa80eeba640 1 Processor -- start 2024-06-06T08:22:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.923+0000 7fa80eeba640 1 -- start start 2024-06-06T08:22:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.923+0000 7fa80eeba640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa808110e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.923+0000 7fa80eeba640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8081113a0 con 0x7fa808101cb0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.923+0000 7fa80cc56640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa808110e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.923+0000 7fa80cc56640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa808110e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43770/0 (socket says 172.21.15.86:43770) 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.923+0000 7fa80cc56640 1 -- 172.21.15.86:0/1913119898 learned_addr learned my addr 172.21.15.86:0/1913119898 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.924+0000 7fa80cc56640 1 -- 172.21.15.86:0/1913119898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc009450 con 0x7fa808101cb0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.924+0000 7fa80cc56640 1 --2- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa808110e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0317b0 tx=0x7fa7fc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.924+0000 7fa8057fa640 1 -- 172.21.15.86:0/1913119898 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7fc031e90 con 0x7fa808101cb0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.924+0000 7fa8057fa640 1 -- 172.21.15.86:0/1913119898 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa7fc03daf0 con 0x7fa808101cb0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.924+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8081115a0 con 0x7fa808101cb0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.924+0000 7fa8057fa640 1 -- 172.21.15.86:0/1913119898 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7fc03baa0 con 0x7fa808101cb0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.924+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa808112080 con 0x7fa808101cb0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.925+0000 7fa8057fa640 1 -- 172.21.15.86:0/1913119898 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fa7fc039070 con 0x7fa808101cb0 2024-06-06T08:22:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.925+0000 7fa8057fa640 1 --2- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa7f8034810 0x7fa7f8036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.925+0000 7fa8057fa640 1 -- 172.21.15.86:0/1913119898 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa7fc067970 con 0x7fa808101cb0 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.925+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80805ead0 con 0x7fa808101cb0 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.925+0000 7fa807fff640 1 --2- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa7f8034810 0x7fa7f8036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.926+0000 7fa807fff640 1 --2- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa7f8034810 0x7fa7f8036cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4006fd0 tx=0x7fa7f4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:00.930+0000 7fa8057fa640 1 -- 172.21.15.86:0/1913119898 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa7fc041080 con 0x7fa808101cb0 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.093+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa808103c20 con 0x7fa7f8034810 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.261+0000 7fa8057fa640 1 -- 172.21.15.86:0/1913119898 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa808103c20 con 0x7fa7f8034810 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.263+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa7f8034810 msgr2=0x7fa7f8036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.263+0000 7fa80eeba640 1 --2- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa7f8034810 0x7fa7f8036cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4006fd0 tx=0x7fa7f4006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.263+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 msgr2=0x7fa808110e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.263+0000 7fa80eeba640 1 --2- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa808110e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0317b0 tx=0x7fa7fc008040 comp rx=0 tx=0).stop 2024-06-06T08:22:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.264+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 shutdown_connections 2024-06-06T08:22:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.264+0000 7fa80eeba640 1 --2- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa7f8034810 0x7fa7f8036cc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.264+0000 7fa80eeba640 1 --2- 172.21.15.86:0/1913119898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa808101cb0 0x7fa808110e60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.264+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 >> 172.21.15.86:0/1913119898 conn(0x7fa8080fd4f0 msgr2=0x7fa8080fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.264+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 shutdown_connections 2024-06-06T08:22:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:01.264+0000 7fa80eeba640 1 -- 172.21.15.86:0/1913119898 wait complete. 2024-06-06T08:22:02.289 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:02 smithi086 ceph-mon[29114]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:02.289 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:02 smithi086 ceph-mon[29114]: Generating ssh key... 2024-06-06T08:22:02.289 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:02 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:02.290 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:02 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDn5CxMUaXjagRRoHiZ059HRFHNprELnTIzIcV4CGpBWpHYfRl05AhOEB3LE587ycrtNy+4gM1SsT5xtPJoL67M+yCPFzRV+/uHnR9aihBVsvKr1YmpMutI5UnBZ4UhOta3ddxNrQ0+zEQ0Sm1H9VviUddbgg456WvhZ6VGWilkW+hWDGxHkc5czGTaPktux132rt0o0pknGJu10YonSFT+LONX7muhEgouoencErbsfBqRYsAFhPEiEXykBlz8XvbR5XmGsA0vGi2j3QrFsQljbyckIRWxFDNDqqbdzkWnt9FVg8XB+1VMc6UGxUcR/AX6WQYBfuv58I3Vttgdw5V0gHzH2JQJQA9z+IFGoYIoZhbBoNKXEtcyPJkCwNYCuKQJv6NYkKxTVSnO2KtNO0YXvHoY11/3z15HDHGs7ERhbutgdoKp/jIla0UQ9Je1gSjL0Zwr2V8CsmvR83YZ0l926w5inuPaIZ+otcSfcXgMf0G4BXzq5Bn4gSDW/hvpUfs= ceph-a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:22:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.110+0000 7fa60d3d3640 1 Processor -- start 2024-06-06T08:22:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.110+0000 7fa60d3d3640 1 -- start start 2024-06-06T08:22:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.110+0000 7fa60d3d3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608102120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.110+0000 7fa60d3d3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa608102660 con 0x7fa608101d10 2024-06-06T08:22:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.110+0000 7fa606ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.110+0000 7fa606ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608102120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43776/0 (socket says 172.21.15.86:43776) 2024-06-06T08:22:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.110+0000 7fa606ffd640 1 -- 172.21.15.86:0/1025282698 learned_addr learned my addr 172.21.15.86:0/1025282698 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.111+0000 7fa606ffd640 1 -- 172.21.15.86:0/1025282698 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6081027a0 con 0x7fa608101d10 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.111+0000 7fa606ffd640 1 --2- 172.21.15.86:0/1025282698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608102120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009840 tx=0x7fa5f80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755d6a343a5b514f server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.111+0000 7fa6067fc640 1 -- 172.21.15.86:0/1025282698 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f8032440 con 0x7fa608101d10 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.111+0000 7fa6067fc640 1 -- 172.21.15.86:0/1025282698 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa5f8032a30 con 0x7fa608101d10 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.111+0000 7fa6067fc640 1 -- 172.21.15.86:0/1025282698 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f8031ad0 con 0x7fa608101d10 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.112+0000 7fa60d3d3640 1 -- 172.21.15.86:0/1025282698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 msgr2=0x7fa608102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.112+0000 7fa60d3d3640 1 --2- 172.21.15.86:0/1025282698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608102120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009840 tx=0x7fa5f80091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.112+0000 7fa60d3d3640 1 -- 172.21.15.86:0/1025282698 shutdown_connections 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.112+0000 7fa60d3d3640 1 --2- 172.21.15.86:0/1025282698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608102120 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.112+0000 7fa60d3d3640 1 -- 172.21.15.86:0/1025282698 >> 172.21.15.86:0/1025282698 conn(0x7fa6080fd4f0 msgr2=0x7fa6080ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.112+0000 7fa60d3d3640 1 -- 172.21.15.86:0/1025282698 shutdown_connections 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.112+0000 7fa60d3d3640 1 -- 172.21.15.86:0/1025282698 wait complete. 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.112+0000 7fa60d3d3640 1 Processor -- start 2024-06-06T08:22:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa60d3d3640 1 -- start start 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa60d3d3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608110e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa60d3d3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081113b0 con 0x7fa608101d10 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa606ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608110e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa606ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608110e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43782/0 (socket says 172.21.15.86:43782) 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa606ffd640 1 -- 172.21.15.86:0/3832846946 learned_addr learned my addr 172.21.15.86:0/3832846946 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa606ffd640 1 -- 172.21.15.86:0/3832846946 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f8009450 con 0x7fa608101d10 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa606ffd640 1 --2- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608110e70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009420 tx=0x7fa5f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa5fffff640 1 -- 172.21.15.86:0/3832846946 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f803d5b0 con 0x7fa608101d10 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa5fffff640 1 -- 172.21.15.86:0/3832846946 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa5f803dba0 con 0x7fa608101d10 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.113+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6081115b0 con 0x7fa608101d10 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.114+0000 7fa5fffff640 1 -- 172.21.15.86:0/3832846946 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f803bb40 con 0x7fa608101d10 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.114+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa608112040 con 0x7fa608101d10 2024-06-06T08:22:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.114+0000 7fa5fffff640 1 -- 172.21.15.86:0/3832846946 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fa5f8039070 con 0x7fa608101d10 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.114+0000 7fa5fffff640 1 --2- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa5f4034860 0x7fa5f4036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.115+0000 7fa5fffff640 1 -- 172.21.15.86:0/3832846946 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa5f80677c0 con 0x7fa608101d10 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.115+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60805ead0 con 0x7fa608101d10 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.115+0000 7fa5fe7fc640 1 --2- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa5f4034860 0x7fa5f4036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.115+0000 7fa5fe7fc640 1 --2- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa5f4034860 0x7fa5f4036d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0006fd0 tx=0x7fa5f0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.119+0000 7fa5fffff640 1 -- 172.21.15.86:0/3832846946 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa5f8041080 con 0x7fa608101d10 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.290+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa608112260 con 0x7fa5f4034860 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.290+0000 7fa5fffff640 1 -- 172.21.15.86:0/3832846946 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa608112260 con 0x7fa5f4034860 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa5f4034860 msgr2=0x7fa5f4036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 --2- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa5f4034860 0x7fa5f4036d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0006fd0 tx=0x7fa5f0006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 msgr2=0x7fa608110e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 --2- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608110e70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009420 tx=0x7fa5f8008040 comp rx=0 tx=0).stop 2024-06-06T08:22:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 shutdown_connections 2024-06-06T08:22:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 --2- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fa5f4034860 0x7fa5f4036d10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 --2- 172.21.15.86:0/3832846946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa608101d10 0x7fa608110e70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 >> 172.21.15.86:0/3832846946 conn(0x7fa6080fd4f0 msgr2=0x7fa6080fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 shutdown_connections 2024-06-06T08:22:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:02.293+0000 7fa60d3d3640 1 -- 172.21.15.86:0/3832846946 wait complete. 2024-06-06T08:22:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-06T08:22:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:Adding key to root@localhost authorized_keys... 2024-06-06T08:22:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:Adding host smithi086... 2024-06-06T08:22:03.468 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:03 smithi086 ceph-mon[29114]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:04.527 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:04 smithi086 ceph-mon[29114]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "addr": "172.21.15.86", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:05.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:05 smithi086 ceph-mon[29114]: Deploying cephadm binary to smithi086 2024-06-06T08:22:06.828 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Added host 'smithi086' with addr '172.21.15.86' 2024-06-06T08:22:06.828 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.289+0000 7f5ca7396640 1 Processor -- start 2024-06-06T08:22:06.828 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.290+0000 7f5ca7396640 1 -- start start 2024-06-06T08:22:06.828 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.290+0000 7f5ca7396640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca01020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.290+0000 7f5ca7396640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca0102600 con 0x7f5ca0101cb0 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.290+0000 7f5ca5132640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca01020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.290+0000 7f5ca5132640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca01020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43790/0 (socket says 172.21.15.86:43790) 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.290+0000 7f5ca5132640 1 -- 172.21.15.86:0/954841325 learned_addr learned my addr 172.21.15.86:0/954841325 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.290+0000 7f5ca5132640 1 -- 172.21.15.86:0/954841325 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca0102740 con 0x7f5ca0101cb0 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.291+0000 7f5ca5132640 1 --2- 172.21.15.86:0/954841325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca01020c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009840 tx=0x7f5c940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ec823ef107512f server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.291+0000 7f5c93fff640 1 -- 172.21.15.86:0/954841325 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c94032440 con 0x7f5ca0101cb0 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.291+0000 7f5c93fff640 1 -- 172.21.15.86:0/954841325 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5c94032a30 con 0x7f5ca0101cb0 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.291+0000 7f5c93fff640 1 -- 172.21.15.86:0/954841325 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c94031ad0 con 0x7f5ca0101cb0 2024-06-06T08:22:06.829 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.292+0000 7f5ca7396640 1 -- 172.21.15.86:0/954841325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 msgr2=0x7f5ca01020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.292+0000 7f5ca7396640 1 --2- 172.21.15.86:0/954841325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca01020c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009840 tx=0x7f5c940091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.292+0000 7f5ca7396640 1 -- 172.21.15.86:0/954841325 shutdown_connections 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.292+0000 7f5ca7396640 1 --2- 172.21.15.86:0/954841325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca01020c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.292+0000 7f5ca7396640 1 -- 172.21.15.86:0/954841325 >> 172.21.15.86:0/954841325 conn(0x7f5ca00fd490 msgr2=0x7f5ca00ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.292+0000 7f5ca7396640 1 -- 172.21.15.86:0/954841325 shutdown_connections 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.292+0000 7f5ca7396640 1 -- 172.21.15.86:0/954841325 wait complete. 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.292+0000 7f5ca7396640 1 Processor -- start 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5ca7396640 1 -- start start 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5ca7396640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca0077bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5ca7396640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca0078100 con 0x7f5ca0101cb0 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5ca5132640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca0077bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5ca5132640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca0077bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43802/0 (socket says 172.21.15.86:43802) 2024-06-06T08:22:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5ca5132640 1 -- 172.21.15.86:0/1573075121 learned_addr learned my addr 172.21.15.86:0/1573075121 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5ca5132640 1 -- 172.21.15.86:0/1573075121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c94009450 con 0x7f5ca0101cb0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5ca5132640 1 --2- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca0077bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009420 tx=0x7f5c94008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5c91ffb640 1 -- 172.21.15.86:0/1573075121 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c9403d5b0 con 0x7f5ca0101cb0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5c91ffb640 1 -- 172.21.15.86:0/1573075121 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5c9403dba0 con 0x7f5ca0101cb0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca00761e0 con 0x7f5ca0101cb0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.293+0000 7f5c91ffb640 1 -- 172.21.15.86:0/1573075121 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c9403bb40 con 0x7f5ca0101cb0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.294+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca0076650 con 0x7f5ca0101cb0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.294+0000 7f5c91ffb640 1 -- 172.21.15.86:0/1573075121 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f5c94039070 con 0x7f5ca0101cb0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.295+0000 7f5c91ffb640 1 --2- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5c8c034810 0x7f5c8c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.295+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca005ead0 con 0x7f5ca0101cb0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.295+0000 7f5c91ffb640 1 -- 172.21.15.86:0/1573075121 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5c940677f0 con 0x7f5ca0101cb0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.295+0000 7f5ca4931640 1 --2- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5c8c034810 0x7f5c8c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:06.831 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.295+0000 7f5ca4931640 1 --2- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5c8c034810 0x7f5c8c036cc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c006fd0 tx=0x7f5c9c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.299+0000 7f5c91ffb640 1 -- 172.21.15.86:0/1573075121 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5c940432c0 con 0x7f5ca0101cb0 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:03.469+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi086", "addr": "172.21.15.86", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ca0076950 con 0x7f5c8c034810 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.250+0000 7f5c91ffb640 1 -- 172.21.15.86:0/1573075121 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f5ca0076950 con 0x7f5c8c034810 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5c8c034810 msgr2=0x7f5c8c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 --2- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5c8c034810 0x7f5c8c036cc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c006fd0 tx=0x7f5c9c006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 msgr2=0x7f5ca0077bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 --2- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca0077bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009420 tx=0x7f5c94008040 comp rx=0 tx=0).stop 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 shutdown_connections 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 --2- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5c8c034810 0x7f5c8c036cc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 --2- 172.21.15.86:0/1573075121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca0101cb0 0x7f5ca0077bc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 >> 172.21.15.86:0/1573075121 conn(0x7f5ca00fd490 msgr2=0x7f5ca00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 shutdown_connections 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:06.253+0000 7f5ca7396640 1 -- 172.21.15.86:0/1573075121 wait complete. 2024-06-06T08:22:06.832 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying mon service with default placement... 2024-06-06T08:22:07.441 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:07 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:07.441 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:07 smithi086 ceph-mon[29114]: Added host smithi086 2024-06-06T08:22:07.441 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:07 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:22:08.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-06T08:22:08.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.260+0000 7f82e8fec640 1 Processor -- start 2024-06-06T08:22:08.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.260+0000 7f82e8fec640 1 -- start start 2024-06-06T08:22:08.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.261+0000 7f82e8fec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41049c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:08.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.261+0000 7f82e8fec640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e4075f30 con 0x7f82e4075b20 2024-06-06T08:22:08.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.261+0000 7f82e259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:08.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.261+0000 7f82e259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41049c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58498/0 (socket says 172.21.15.86:58498) 2024-06-06T08:22:08.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.261+0000 7f82e259c640 1 -- 172.21.15.86:0/3615485379 learned_addr learned my addr 172.21.15.86:0/3615485379 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:08.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.261+0000 7f82e259c640 1 -- 172.21.15.86:0/3615485379 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e4104f00 con 0x7f82e4075b20 2024-06-06T08:22:08.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.262+0000 7f82e259c640 1 --2- 172.21.15.86:0/3615485379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41049c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009e50 tx=0x7f82d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6dc403af08bfdc server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:08.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.262+0000 7f82e159a640 1 -- 172.21.15.86:0/3615485379 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82d4031560 con 0x7f82e4075b20 2024-06-06T08:22:08.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.262+0000 7f82e159a640 1 -- 172.21.15.86:0/3615485379 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f82d402d050 con 0x7f82e4075b20 2024-06-06T08:22:08.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.262+0000 7f82e159a640 1 -- 172.21.15.86:0/3615485379 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82d4007a50 con 0x7f82e4075b20 2024-06-06T08:22:08.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.262+0000 7f82e8fec640 1 -- 172.21.15.86:0/3615485379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 msgr2=0x7f82e41049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:08.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.262+0000 7f82e8fec640 1 --2- 172.21.15.86:0/3615485379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41049c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009e50 tx=0x7f82d4009840 comp rx=0 tx=0).stop 2024-06-06T08:22:08.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e8fec640 1 -- 172.21.15.86:0/3615485379 shutdown_connections 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e8fec640 1 --2- 172.21.15.86:0/3615485379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41049c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e8fec640 1 -- 172.21.15.86:0/3615485379 >> 172.21.15.86:0/3615485379 conn(0x7f82e40fc430 msgr2=0x7f82e40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e8fec640 1 -- 172.21.15.86:0/3615485379 shutdown_connections 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e8fec640 1 -- 172.21.15.86:0/3615485379 wait complete. 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e8fec640 1 Processor -- start 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e8fec640 1 -- start start 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e8fec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41022f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e8fec640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e4102830 con 0x7f82e4075b20 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41022f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41022f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58500/0 (socket says 172.21.15.86:58500) 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.263+0000 7f82e259c640 1 -- 172.21.15.86:0/2913377224 learned_addr learned my addr 172.21.15.86:0/2913377224 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.264+0000 7f82e259c640 1 -- 172.21.15.86:0/2913377224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d4009ae0 con 0x7f82e4075b20 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.264+0000 7f82e259c640 1 --2- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41022f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f82d4007030 tx=0x7f82d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.264+0000 7f82daffd640 1 -- 172.21.15.86:0/2913377224 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82d4031dc0 con 0x7f82e4075b20 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.264+0000 7f82daffd640 1 -- 172.21.15.86:0/2913377224 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f82d40458c0 con 0x7f82e4075b20 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.264+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e41008c0 con 0x7f82e4075b20 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.264+0000 7f82daffd640 1 -- 172.21.15.86:0/2913377224 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82d403a8f0 con 0x7f82e4075b20 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.264+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e4100d30 con 0x7f82e4075b20 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.265+0000 7f82daffd640 1 -- 172.21.15.86:0/2913377224 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f82d4038040 con 0x7f82e4075b20 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.265+0000 7f82daffd640 1 --2- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f82d0034860 0x7f82d0036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:08.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.265+0000 7f82daffd640 1 -- 172.21.15.86:0/2913377224 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f82d4040030 con 0x7f82e4075b20 2024-06-06T08:22:08.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.265+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e404b480 con 0x7f82e4075b20 2024-06-06T08:22:08.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.265+0000 7f82e1d9b640 1 --2- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f82d0034860 0x7f82d0036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:08.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.266+0000 7f82e1d9b640 1 --2- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f82d0034860 0x7f82d0036d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f82cc006fd0 tx=0x7f82cc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:08.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.270+0000 7f82daffd640 1 -- 172.21.15.86:0/2913377224 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f82d4036070 con 0x7f82e4075b20 2024-06-06T08:22:08.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.442+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f82e41012b0 con 0x7f82d0034860 2024-06-06T08:22:08.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.449+0000 7f82daffd640 1 -- 172.21.15.86:0/2913377224 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f82e41012b0 con 0x7f82d0034860 2024-06-06T08:22:08.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f82d0034860 msgr2=0x7f82d0036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:08.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 --2- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f82d0034860 0x7f82d0036d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f82cc006fd0 tx=0x7f82cc006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:08.178 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 msgr2=0x7f82e41022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:08.178 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 --2- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41022f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f82d4007030 tx=0x7f82d4009840 comp rx=0 tx=0).stop 2024-06-06T08:22:08.178 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 shutdown_connections 2024-06-06T08:22:08.178 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 --2- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f82d0034860 0x7f82d0036d10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:08.178 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 --2- 172.21.15.86:0/2913377224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e4075b20 0x7f82e41022f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:08.178 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 >> 172.21.15.86:0/2913377224 conn(0x7f82e40fc430 msgr2=0x7f82e40fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:08.178 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 shutdown_connections 2024-06-06T08:22:08.178 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:07.451+0000 7f82e8fec640 1 -- 172.21.15.86:0/2913377224 wait complete. 2024-06-06T08:22:08.178 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying mgr service with default placement... 2024-06-06T08:22:08.729 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:08 smithi086 ceph-mon[29114]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:08.729 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:08 smithi086 ceph-mon[29114]: Saving service mon spec with placement count:5 2024-06-06T08:22:08.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:08 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.541+0000 7f22e7ea8640 1 Processor -- start 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.542+0000 7f22e7ea8640 1 -- start start 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.542+0000 7f22e7ea8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e01004d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.542+0000 7f22e7ea8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0100a10 con 0x7f22e01000c0 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.542+0000 7f22e5c44640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e01004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.542+0000 7f22e5c44640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e01004d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58502/0 (socket says 172.21.15.86:58502) 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.542+0000 7f22e5c44640 1 -- 172.21.15.86:0/3945502557 learned_addr learned my addr 172.21.15.86:0/3945502557 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.542+0000 7f22e5c44640 1 -- 172.21.15.86:0/3945502557 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e0100b50 con 0x7f22e01000c0 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.543+0000 7f22e5c44640 1 --2- 172.21.15.86:0/3945502557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e01004d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009840 tx=0x7f22dc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577ab2cf12f8fd2a server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:09.373 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.543+0000 7f22e4c42640 1 -- 172.21.15.86:0/3945502557 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22dc032440 con 0x7f22e01000c0 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.543+0000 7f22e4c42640 1 -- 172.21.15.86:0/3945502557 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f22dc032a30 con 0x7f22e01000c0 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.543+0000 7f22e4c42640 1 -- 172.21.15.86:0/3945502557 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22dc031ad0 con 0x7f22e01000c0 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.543+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3945502557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 msgr2=0x7f22e01004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.543+0000 7f22e7ea8640 1 --2- 172.21.15.86:0/3945502557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e01004d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009840 tx=0x7f22dc0091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3945502557 shutdown_connections 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e7ea8640 1 --2- 172.21.15.86:0/3945502557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e01004d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3945502557 >> 172.21.15.86:0/3945502557 conn(0x7f22e00fbbc0 msgr2=0x7f22e00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3945502557 shutdown_connections 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3945502557 wait complete. 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e7ea8640 1 Processor -- start 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e7ea8640 1 -- start start 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e7ea8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e0110ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:09.374 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e7ea8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0111000 con 0x7f22e01000c0 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e5c44640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e0110ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e5c44640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e0110ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58516/0 (socket says 172.21.15.86:58516) 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.544+0000 7f22e5c44640 1 -- 172.21.15.86:0/3203302386 learned_addr learned my addr 172.21.15.86:0/3203302386 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.545+0000 7f22e5c44640 1 -- 172.21.15.86:0/3203302386 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc009450 con 0x7f22e01000c0 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.545+0000 7f22e5c44640 1 --2- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e0110ac0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f22dc0317b0 tx=0x7f22dc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.545+0000 7f22d67fc640 1 -- 172.21.15.86:0/3203302386 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22dc031e90 con 0x7f22e01000c0 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.545+0000 7f22d67fc640 1 -- 172.21.15.86:0/3203302386 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f22dc03daf0 con 0x7f22e01000c0 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.545+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e0111200 con 0x7f22e01000c0 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.545+0000 7f22d67fc640 1 -- 172.21.15.86:0/3203302386 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22dc03baa0 con 0x7f22e01000c0 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.545+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e0111670 con 0x7f22e01000c0 2024-06-06T08:22:09.375 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.546+0000 7f22d67fc640 1 -- 172.21.15.86:0/3203302386 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f22dc039070 con 0x7f22e01000c0 2024-06-06T08:22:09.376 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.546+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e004b480 con 0x7f22e01000c0 2024-06-06T08:22:09.376 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.546+0000 7f22d67fc640 1 --2- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f22cc034810 0x7f22cc036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:09.376 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.546+0000 7f22d67fc640 1 -- 172.21.15.86:0/3203302386 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f22dc067740 con 0x7f22e01000c0 2024-06-06T08:22:09.376 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.546+0000 7f22e5443640 1 --2- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f22cc034810 0x7f22cc036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:09.376 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.547+0000 7f22e5443640 1 --2- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f22cc034810 0x7f22cc036cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22d0006fd0 tx=0x7f22d0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:09.376 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.550+0000 7f22d67fc640 1 -- 172.21.15.86:0/3203302386 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f22dc03ca00 con 0x7f22e01000c0 2024-06-06T08:22:09.376 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.729+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f22e0111670 con 0x7f22cc034810 2024-06-06T08:22:09.376 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.733+0000 7f22d67fc640 1 -- 172.21.15.86:0/3203302386 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f22e0111670 con 0x7f22cc034810 2024-06-06T08:22:09.377 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f22cc034810 msgr2=0x7f22cc036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:09.377 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 --2- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f22cc034810 0x7f22cc036cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22d0006fd0 tx=0x7f22d0006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:09.377 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 msgr2=0x7f22e0110ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:09.377 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 --2- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e0110ac0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f22dc0317b0 tx=0x7f22dc008040 comp rx=0 tx=0).stop 2024-06-06T08:22:09.377 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 shutdown_connections 2024-06-06T08:22:09.377 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 --2- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f22cc034810 0x7f22cc036cc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:09.377 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 --2- 172.21.15.86:0/3203302386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01000c0 0x7f22e0110ac0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:09.377 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 >> 172.21.15.86:0/3203302386 conn(0x7f22e00fbbc0 msgr2=0x7f22e00fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:09.378 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 shutdown_connections 2024-06-06T08:22:09.378 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:08.735+0000 7f22e7ea8640 1 -- 172.21.15.86:0/3203302386 wait complete. 2024-06-06T08:22:09.378 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying crash service with default placement... 2024-06-06T08:22:09.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:09 smithi086 ceph-mon[29114]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:09.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:09 smithi086 ceph-mon[29114]: Saving service mgr spec with placement count:2 2024-06-06T08:22:09.910 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:09 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:11.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:10 smithi086 ceph-mon[29114]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:11.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:10 smithi086 ceph-mon[29114]: Saving service crash spec with placement * 2024-06-06T08:22:11.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:10 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:11.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:10 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:11.051 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-06-06T08:22:11.056 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.724+0000 7f5fd0e33640 1 Processor -- start 2024-06-06T08:22:11.056 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.724+0000 7f5fd0e33640 1 -- start start 2024-06-06T08:22:11.056 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.724+0000 7f5fd0e33640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc1020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:11.056 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.724+0000 7f5fd0e33640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc102600 con 0x7f5fcc101cb0 2024-06-06T08:22:11.056 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.724+0000 7f5fca59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc1020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.724+0000 7f5fca59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc1020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58518/0 (socket says 172.21.15.86:58518) 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.724+0000 7f5fca59c640 1 -- 172.21.15.86:0/4124329430 learned_addr learned my addr 172.21.15.86:0/4124329430 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.724+0000 7f5fca59c640 1 -- 172.21.15.86:0/4124329430 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc102740 con 0x7f5fcc101cb0 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.725+0000 7f5fca59c640 1 --2- 172.21.15.86:0/4124329430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc1020c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009840 tx=0x7f5fbc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03a221970444df3 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.725+0000 7f5fc9d9b640 1 -- 172.21.15.86:0/4124329430 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fbc032440 con 0x7f5fcc101cb0 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.725+0000 7f5fc9d9b640 1 -- 172.21.15.86:0/4124329430 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5fbc032a30 con 0x7f5fcc101cb0 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.725+0000 7f5fc9d9b640 1 -- 172.21.15.86:0/4124329430 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fbc031ad0 con 0x7f5fcc101cb0 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.726+0000 7f5fd0e33640 1 -- 172.21.15.86:0/4124329430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 msgr2=0x7f5fcc1020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.726+0000 7f5fd0e33640 1 --2- 172.21.15.86:0/4124329430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc1020c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009840 tx=0x7f5fbc0091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.726+0000 7f5fd0e33640 1 -- 172.21.15.86:0/4124329430 shutdown_connections 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.726+0000 7f5fd0e33640 1 --2- 172.21.15.86:0/4124329430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc1020c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.726+0000 7f5fd0e33640 1 -- 172.21.15.86:0/4124329430 >> 172.21.15.86:0/4124329430 conn(0x7f5fcc0fd490 msgr2=0x7f5fcc0ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:11.057 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.726+0000 7f5fd0e33640 1 -- 172.21.15.86:0/4124329430 shutdown_connections 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.726+0000 7f5fd0e33640 1 -- 172.21.15.86:0/4124329430 wait complete. 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.726+0000 7f5fd0e33640 1 Processor -- start 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.726+0000 7f5fd0e33640 1 -- start start 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fd0e33640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc110eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fd0e33640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc1113f0 con 0x7f5fcc101cb0 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fca59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc110eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fca59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc110eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58522/0 (socket says 172.21.15.86:58522) 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fca59c640 1 -- 172.21.15.86:0/3134312322 learned_addr learned my addr 172.21.15.86:0/3134312322 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fca59c640 1 -- 172.21.15.86:0/3134312322 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc009450 con 0x7f5fcc101cb0 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fca59c640 1 --2- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc110eb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009420 tx=0x7f5fbc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fc2ffd640 1 -- 172.21.15.86:0/3134312322 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fbc03d5b0 con 0x7f5fcc101cb0 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fc2ffd640 1 -- 172.21.15.86:0/3134312322 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5fbc03dba0 con 0x7f5fcc101cb0 2024-06-06T08:22:11.058 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fcc1115f0 con 0x7f5fcc101cb0 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fc2ffd640 1 -- 172.21.15.86:0/3134312322 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fbc03bb40 con 0x7f5fcc101cb0 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.727+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fcc111ff0 con 0x7f5fcc101cb0 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.728+0000 7f5fc2ffd640 1 -- 172.21.15.86:0/3134312322 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f5fbc039070 con 0x7f5fcc101cb0 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.728+0000 7f5fc2ffd640 1 --2- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5fb8034860 0x7f5fb8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.728+0000 7f5fc2ffd640 1 -- 172.21.15.86:0/3134312322 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5fbc0679a0 con 0x7f5fcc101cb0 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.728+0000 7f5fc3fff640 1 --2- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5fb8034860 0x7f5fb8036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.728+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fcc05ead0 con 0x7f5fcc101cb0 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.729+0000 7f5fc3fff640 1 --2- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5fb8034860 0x7f5fb8036d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4006fd0 tx=0x7f5fb4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.733+0000 7f5fc2ffd640 1 -- 172.21.15.86:0/3134312322 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5fbc041070 con 0x7f5fcc101cb0 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.909+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fcc103c20 con 0x7f5fb8034860 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.916+0000 7f5fc2ffd640 1 -- 172.21.15.86:0/3134312322 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f5fcc103c20 con 0x7f5fb8034860 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5fb8034860 msgr2=0x7f5fb8036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:11.059 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 --2- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5fb8034860 0x7f5fb8036d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4006fd0 tx=0x7f5fb4006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:11.060 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 msgr2=0x7f5fcc110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:11.060 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 --2- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc110eb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009420 tx=0x7f5fbc008040 comp rx=0 tx=0).stop 2024-06-06T08:22:11.060 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 shutdown_connections 2024-06-06T08:22:11.060 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 --2- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f5fb8034860 0x7f5fb8036d10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:11.060 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 --2- 172.21.15.86:0/3134312322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fcc101cb0 0x7f5fcc110eb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:11.060 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 >> 172.21.15.86:0/3134312322 conn(0x7f5fcc0fd490 msgr2=0x7f5fcc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:11.060 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 shutdown_connections 2024-06-06T08:22:11.060 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:09.918+0000 7f5fd0e33640 1 -- 172.21.15.86:0/3134312322 wait complete. 2024-06-06T08:22:11.060 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying prometheus service with default placement... 2024-06-06T08:22:12.138 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-06-06T08:22:12.138 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.412+0000 7fb352ee3640 1 Processor -- start 2024-06-06T08:22:12.138 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.412+0000 7fb352ee3640 1 -- start start 2024-06-06T08:22:12.138 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.413+0000 7fb352ee3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:12.138 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.413+0000 7fb352ee3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c075f70 con 0x7fb34c075b60 2024-06-06T08:22:12.138 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.413+0000 7fb350c7f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:12.138 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.413+0000 7fb350c7f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58532/0 (socket says 172.21.15.86:58532) 2024-06-06T08:22:12.138 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.413+0000 7fb350c7f640 1 -- 172.21.15.86:0/203448849 learned_addr learned my addr 172.21.15.86:0/203448849 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:12.138 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.413+0000 7fb350c7f640 1 -- 172.21.15.86:0/203448849 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c104ec0 con 0x7fb34c075b60 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.414+0000 7fb350c7f640 1 --2- 172.21.15.86:0/203448849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c104980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb340009840 tx=0x7fb3400091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6145cdccd6b5af7 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.414+0000 7fb34b7fe640 1 -- 172.21.15.86:0/203448849 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb340032440 con 0x7fb34c075b60 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.414+0000 7fb34b7fe640 1 -- 172.21.15.86:0/203448849 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb340032a70 con 0x7fb34c075b60 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.414+0000 7fb34b7fe640 1 -- 172.21.15.86:0/203448849 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb340031ad0 con 0x7fb34c075b60 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.414+0000 7fb352ee3640 1 -- 172.21.15.86:0/203448849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 msgr2=0x7fb34c104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.414+0000 7fb352ee3640 1 --2- 172.21.15.86:0/203448849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c104980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb340009840 tx=0x7fb3400091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.415+0000 7fb352ee3640 1 -- 172.21.15.86:0/203448849 shutdown_connections 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.415+0000 7fb352ee3640 1 --2- 172.21.15.86:0/203448849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c104980 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.415+0000 7fb352ee3640 1 -- 172.21.15.86:0/203448849 >> 172.21.15.86:0/203448849 conn(0x7fb34c0fc430 msgr2=0x7fb34c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.415+0000 7fb352ee3640 1 -- 172.21.15.86:0/203448849 shutdown_connections 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.415+0000 7fb352ee3640 1 -- 172.21.15.86:0/203448849 wait complete. 2024-06-06T08:22:12.139 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.415+0000 7fb352ee3640 1 Processor -- start 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.415+0000 7fb352ee3640 1 -- start start 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.415+0000 7fb352ee3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.415+0000 7fb352ee3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c111110 con 0x7fb34c075b60 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb350c7f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb350c7f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58548/0 (socket says 172.21.15.86:58548) 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb350c7f640 1 -- 172.21.15.86:0/2884603863 learned_addr learned my addr 172.21.15.86:0/2884603863 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb350c7f640 1 -- 172.21.15.86:0/2884603863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb340009450 con 0x7fb34c075b60 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb350c7f640 1 --2- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c110bd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb3400098a0 tx=0x7fb34002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb3497fa640 1 -- 172.21.15.86:0/2884603863 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb340031e90 con 0x7fb34c075b60 2024-06-06T08:22:12.140 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb3497fa640 1 -- 172.21.15.86:0/2884603863 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb34003b910 con 0x7fb34c075b60 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c111310 con 0x7fb34c075b60 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb3497fa640 1 -- 172.21.15.86:0/2884603863 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb34003a9b0 con 0x7fb34c075b60 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.416+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c111780 con 0x7fb34c075b60 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.417+0000 7fb3497fa640 1 -- 172.21.15.86:0/2884603863 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fb340039070 con 0x7fb34c075b60 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.417+0000 7fb3497fa640 1 --2- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb33c034810 0x7fb33c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.417+0000 7fb3497fa640 1 -- 172.21.15.86:0/2884603863 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb340067d60 con 0x7fb34c075b60 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.417+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c111780 con 0x7fb34c075b60 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.417+0000 7fb34bfff640 1 --2- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb33c034810 0x7fb33c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.418+0000 7fb34bfff640 1 --2- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb33c034810 0x7fb33c036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb338006fd0 tx=0x7fb338006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.426+0000 7fb3497fa640 1 -- 172.21.15.86:0/2884603863 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb34c111780 con 0x7fb34c075b60 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.626+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fb34c111780 con 0x7fb33c034810 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.632+0000 7fb3497fa640 1 -- 172.21.15.86:0/2884603863 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fb34c111780 con 0x7fb33c034810 2024-06-06T08:22:12.141 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb33c034810 msgr2=0x7fb33c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 --2- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb33c034810 0x7fb33c036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb338006fd0 tx=0x7fb338006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 msgr2=0x7fb34c110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 --2- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c110bd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb3400098a0 tx=0x7fb34002c040 comp rx=0 tx=0).stop 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 shutdown_connections 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 --2- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb33c034810 0x7fb33c036cc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 --2- 172.21.15.86:0/2884603863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34c075b60 0x7fb34c110bd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 >> 172.21.15.86:0/2884603863 conn(0x7fb34c0fc430 msgr2=0x7fb34c0fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 shutdown_connections 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:11.634+0000 7fb352ee3640 1 -- 172.21.15.86:0/2884603863 wait complete. 2024-06-06T08:22:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying grafana service with default placement... 2024-06-06T08:22:12.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:12 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:12.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:12 smithi086 ceph-mon[29114]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:12.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:12 smithi086 ceph-mon[29114]: Saving service prometheus spec with placement count:1 2024-06-06T08:22:12.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:12 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:13.504 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-06-06T08:22:13.504 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.532+0000 7f8cd719b640 1 Processor -- start 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.532+0000 7f8cd719b640 1 -- start start 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.533+0000 7f8cd719b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd01010c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.533+0000 7f8cd719b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0101600 con 0x7f8cd0100cb0 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.533+0000 7f8cd4f37640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd01010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.533+0000 7f8cd4f37640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd01010c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58562/0 (socket says 172.21.15.86:58562) 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.533+0000 7f8cd4f37640 1 -- 172.21.15.86:0/2806883898 learned_addr learned my addr 172.21.15.86:0/2806883898 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.533+0000 7f8cd4f37640 1 -- 172.21.15.86:0/2806883898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd0101740 con 0x7f8cd0100cb0 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.534+0000 7f8cd4f37640 1 --2- 172.21.15.86:0/2806883898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd01010c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009840 tx=0x7f8ccc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf1ba78904c3152 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.534+0000 7f8cc77fe640 1 -- 172.21.15.86:0/2806883898 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ccc032440 con 0x7f8cd0100cb0 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.534+0000 7f8cc77fe640 1 -- 172.21.15.86:0/2806883898 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ccc032a70 con 0x7f8cd0100cb0 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.534+0000 7f8cc77fe640 1 -- 172.21.15.86:0/2806883898 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ccc031ad0 con 0x7f8cd0100cb0 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.534+0000 7f8cd719b640 1 -- 172.21.15.86:0/2806883898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 msgr2=0x7f8cd01010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:13.505 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.534+0000 7f8cd719b640 1 --2- 172.21.15.86:0/2806883898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd01010c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009840 tx=0x7f8ccc0091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:13.506 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.534+0000 7f8cd719b640 1 -- 172.21.15.86:0/2806883898 shutdown_connections 2024-06-06T08:22:13.506 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.534+0000 7f8cd719b640 1 --2- 172.21.15.86:0/2806883898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd01010c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:13.506 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.534+0000 7f8cd719b640 1 -- 172.21.15.86:0/2806883898 >> 172.21.15.86:0/2806883898 conn(0x7f8cd00fc490 msgr2=0x7f8cd00fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:13.506 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.535+0000 7f8cd719b640 1 -- 172.21.15.86:0/2806883898 shutdown_connections 2024-06-06T08:22:13.506 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.535+0000 7f8cd719b640 1 -- 172.21.15.86:0/2806883898 wait complete. 2024-06-06T08:22:13.506 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.535+0000 7f8cd719b640 1 Processor -- start 2024-06-06T08:22:13.506 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.535+0000 7f8cd719b640 1 -- start start 2024-06-06T08:22:13.506 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.535+0000 7f8cd719b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd0110b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:13.506 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.535+0000 7f8cd719b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01110a0 con 0x7f8cd0100cb0 2024-06-06T08:22:13.507 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.535+0000 7f8cd4f37640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd0110b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:13.507 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.535+0000 7f8cd4f37640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd0110b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58572/0 (socket says 172.21.15.86:58572) 2024-06-06T08:22:13.507 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.535+0000 7f8cd4f37640 1 -- 172.21.15.86:0/3609821503 learned_addr learned my addr 172.21.15.86:0/3609821503 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:13.507 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.536+0000 7f8cd4f37640 1 -- 172.21.15.86:0/3609821503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc009450 con 0x7f8cd0100cb0 2024-06-06T08:22:13.507 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.536+0000 7f8cd4f37640 1 --2- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd0110b60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc006a70 tx=0x7f8ccc02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:13.507 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.536+0000 7f8cc57fa640 1 -- 172.21.15.86:0/3609821503 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ccc031e90 con 0x7f8cd0100cb0 2024-06-06T08:22:13.508 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.536+0000 7f8cc57fa640 1 -- 172.21.15.86:0/3609821503 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ccc03b910 con 0x7f8cd0100cb0 2024-06-06T08:22:13.508 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.536+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd01112a0 con 0x7f8cd0100cb0 2024-06-06T08:22:13.508 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.536+0000 7f8cc57fa640 1 -- 172.21.15.86:0/3609821503 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ccc03a940 con 0x7f8cd0100cb0 2024-06-06T08:22:13.508 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.536+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd0111710 con 0x7f8cd0100cb0 2024-06-06T08:22:13.508 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.537+0000 7f8cc57fa640 1 -- 172.21.15.86:0/3609821503 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f8ccc039070 con 0x7f8cd0100cb0 2024-06-06T08:22:13.508 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.537+0000 7f8cc57fa640 1 --2- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8cb8034860 0x7f8cb8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:13.508 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.537+0000 7f8cc57fa640 1 -- 172.21.15.86:0/3609821503 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8ccc066d60 con 0x7f8cd0100cb0 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.537+0000 7f8cc7fff640 1 --2- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8cb8034860 0x7f8cb8036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.537+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd005ead0 con 0x7f8cd0100cb0 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.537+0000 7f8cc7fff640 1 --2- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8cb8034860 0x7f8cb8036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0006fd0 tx=0x7f8cc0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.541+0000 7f8cc57fa640 1 -- 172.21.15.86:0/3609821503 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8ccc03f080 con 0x7f8cd0100cb0 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.735+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cd010a920 con 0x7f8cb8034860 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.740+0000 7f8cc57fa640 1 -- 172.21.15.86:0/3609821503 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8cd010a920 con 0x7f8cb8034860 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8cb8034860 msgr2=0x7f8cb8036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 --2- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8cb8034860 0x7f8cb8036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0006fd0 tx=0x7f8cc0006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 msgr2=0x7f8cd0110b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 --2- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd0110b60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc006a70 tx=0x7f8ccc02c040 comp rx=0 tx=0).stop 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 shutdown_connections 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 --2- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8cb8034860 0x7f8cb8036d10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 --2- 172.21.15.86:0/3609821503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd0100cb0 0x7f8cd0110b60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:13.510 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 >> 172.21.15.86:0/3609821503 conn(0x7f8cd00fc490 msgr2=0x7f8cd00fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:13.510 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 shutdown_connections 2024-06-06T08:22:13.510 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:12.742+0000 7f8cd719b640 1 -- 172.21.15.86:0/3609821503 wait complete. 2024-06-06T08:22:13.510 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying node-exporter service with default placement... 2024-06-06T08:22:13.949 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:13 smithi086 ceph-mon[29114]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:13.949 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:13 smithi086 ceph-mon[29114]: Saving service grafana spec with placement count:1 2024-06-06T08:22:13.949 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:13 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:14.938 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-06-06T08:22:14.938 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.194+0000 7f3d01a13640 1 Processor -- start 2024-06-06T08:22:14.938 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.194+0000 7f3d01a13640 1 -- start start 2024-06-06T08:22:14.938 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.194+0000 7f3d01a13640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:14.938 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.194+0000 7f3d01a13640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc078120 con 0x7f3cfc077d10 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.194+0000 7f3cfaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.194+0000 7f3cfaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58588/0 (socket says 172.21.15.86:58588) 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.194+0000 7f3cfaffd640 1 -- 172.21.15.86:0/2551053941 learned_addr learned my addr 172.21.15.86:0/2551053941 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.194+0000 7f3cfaffd640 1 -- 172.21.15.86:0/2551053941 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc076780 con 0x7f3cfc077d10 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3cfaffd640 1 --2- 172.21.15.86:0/2551053941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc076240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009840 tx=0x7f3cec0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56bcb141a17bda8 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3cf9ffb640 1 -- 172.21.15.86:0/2551053941 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec032440 con 0x7f3cfc077d10 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3cf9ffb640 1 -- 172.21.15.86:0/2551053941 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cec032a70 con 0x7f3cfc077d10 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3cf9ffb640 1 -- 172.21.15.86:0/2551053941 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec031ad0 con 0x7f3cfc077d10 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3d01a13640 1 -- 172.21.15.86:0/2551053941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 msgr2=0x7f3cfc076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3d01a13640 1 --2- 172.21.15.86:0/2551053941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc076240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009840 tx=0x7f3cec0091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3d01a13640 1 -- 172.21.15.86:0/2551053941 shutdown_connections 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3d01a13640 1 --2- 172.21.15.86:0/2551053941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc076240 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3d01a13640 1 -- 172.21.15.86:0/2551053941 >> 172.21.15.86:0/2551053941 conn(0x7f3cfc0fd490 msgr2=0x7f3cfc0ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3d01a13640 1 -- 172.21.15.86:0/2551053941 shutdown_connections 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.195+0000 7f3d01a13640 1 -- 172.21.15.86:0/2551053941 wait complete. 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.196+0000 7f3d01a13640 1 Processor -- start 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.196+0000 7f3d01a13640 1 -- start start 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.196+0000 7f3d01a13640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc1130e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.196+0000 7f3d01a13640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc113620 con 0x7f3cfc077d10 2024-06-06T08:22:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.196+0000 7f3cfaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc1130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.196+0000 7f3cfaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc1130e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58596/0 (socket says 172.21.15.86:58596) 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.196+0000 7f3cfaffd640 1 -- 172.21.15.86:0/2026516999 learned_addr learned my addr 172.21.15.86:0/2026516999 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.197+0000 7f3cfaffd640 1 -- 172.21.15.86:0/2026516999 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec009450 con 0x7f3cfc077d10 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.197+0000 7f3cfaffd640 1 --2- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc1130e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0098a0 tx=0x7f3cec02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.197+0000 7f3ce37fe640 1 -- 172.21.15.86:0/2026516999 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec031e90 con 0x7f3cfc077d10 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.197+0000 7f3ce37fe640 1 -- 172.21.15.86:0/2026516999 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cec03b910 con 0x7f3cfc077d10 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.197+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc113820 con 0x7f3cfc077d10 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.197+0000 7f3ce37fe640 1 -- 172.21.15.86:0/2026516999 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec03a9b0 con 0x7f3cfc077d10 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.197+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc113c90 con 0x7f3cfc077d10 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.198+0000 7f3ce37fe640 1 -- 172.21.15.86:0/2026516999 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f3cec039070 con 0x7f3cfc077d10 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.198+0000 7f3ce37fe640 1 --2- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f3ce4034860 0x7f3ce4036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:14.941 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.198+0000 7f3ce37fe640 1 -- 172.21.15.86:0/2026516999 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3cec066df0 con 0x7f3cfc077d10 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.198+0000 7f3cfa7fc640 1 --2- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f3ce4034860 0x7f3ce4036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.198+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc05ead0 con 0x7f3cfc077d10 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.198+0000 7f3cfa7fc640 1 --2- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f3ce4034860 0x7f3ce4036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0006fd0 tx=0x7f3cf0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.203+0000 7f3ce37fe640 1 -- 172.21.15.86:0/2026516999 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3cec03f080 con 0x7f3cfc077d10 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.398+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cfc108100 con 0x7f3ce4034860 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.402+0000 7f3ce37fe640 1 -- 172.21.15.86:0/2026516999 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f3cfc108100 con 0x7f3ce4034860 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f3ce4034860 msgr2=0x7f3ce4036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 --2- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f3ce4034860 0x7f3ce4036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0006fd0 tx=0x7f3cf0006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 msgr2=0x7f3cfc1130e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:14.942 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 --2- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc1130e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0098a0 tx=0x7f3cec02c040 comp rx=0 tx=0).stop 2024-06-06T08:22:14.943 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 shutdown_connections 2024-06-06T08:22:14.943 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 --2- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f3ce4034860 0x7f3ce4036d10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:14.943 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 --2- 172.21.15.86:0/2026516999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cfc077d10 0x7f3cfc1130e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:14.943 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 >> 172.21.15.86:0/2026516999 conn(0x7f3cfc0fd490 msgr2=0x7f3cfc0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:14.943 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 shutdown_connections 2024-06-06T08:22:14.943 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:14.404+0000 7f3d01a13640 1 -- 172.21.15.86:0/2026516999 wait complete. 2024-06-06T08:22:14.943 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying alertmanager service with default placement... 2024-06-06T08:22:15.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:15 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:15.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:15 smithi086 ceph-mon[29114]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:15.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:15 smithi086 ceph-mon[29114]: Saving service node-exporter spec with placement * 2024-06-06T08:22:15.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:15 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:15.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:15 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:16.218 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-06-06T08:22:16.218 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.346+0000 7fb279343640 1 Processor -- start 2024-06-06T08:22:16.219 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.346+0000 7fb279343640 1 -- start start 2024-06-06T08:22:16.219 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.346+0000 7fb279343640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb274076200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:16.219 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.346+0000 7fb279343640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740780e0 con 0x7fb274077cd0 2024-06-06T08:22:16.219 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.346+0000 7fb272ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb274076200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:16.219 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.346+0000 7fb272ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb274076200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58604/0 (socket says 172.21.15.86:58604) 2024-06-06T08:22:16.219 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.346+0000 7fb272ffd640 1 -- 172.21.15.86:0/2945641926 learned_addr learned my addr 172.21.15.86:0/2945641926 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:16.219 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.346+0000 7fb272ffd640 1 -- 172.21.15.86:0/2945641926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274076740 con 0x7fb274077cd0 2024-06-06T08:22:16.219 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.347+0000 7fb272ffd640 1 --2- 172.21.15.86:0/2945641926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb274076200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb264009840 tx=0x7fb2640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a6d1b5ed8a3ad server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:16.219 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.347+0000 7fb271ffb640 1 -- 172.21.15.86:0/2945641926 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb264032440 con 0x7fb274077cd0 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.347+0000 7fb271ffb640 1 -- 172.21.15.86:0/2945641926 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb264032a70 con 0x7fb274077cd0 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.347+0000 7fb271ffb640 1 -- 172.21.15.86:0/2945641926 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb264031ad0 con 0x7fb274077cd0 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.348+0000 7fb279343640 1 -- 172.21.15.86:0/2945641926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 msgr2=0x7fb274076200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.348+0000 7fb279343640 1 --2- 172.21.15.86:0/2945641926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb274076200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb264009840 tx=0x7fb2640091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.348+0000 7fb279343640 1 -- 172.21.15.86:0/2945641926 shutdown_connections 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.348+0000 7fb279343640 1 --2- 172.21.15.86:0/2945641926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb274076200 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.348+0000 7fb279343640 1 -- 172.21.15.86:0/2945641926 >> 172.21.15.86:0/2945641926 conn(0x7fb2740fd490 msgr2=0x7fb2740ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.348+0000 7fb279343640 1 -- 172.21.15.86:0/2945641926 shutdown_connections 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.348+0000 7fb279343640 1 -- 172.21.15.86:0/2945641926 wait complete. 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb279343640 1 Processor -- start 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb279343640 1 -- start start 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb279343640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb2741130d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:16.220 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb279343640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb274113610 con 0x7fb274077cd0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb272ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb2741130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb272ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb2741130d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58612/0 (socket says 172.21.15.86:58612) 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb272ffd640 1 -- 172.21.15.86:0/22280154 learned_addr learned my addr 172.21.15.86:0/22280154 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb272ffd640 1 -- 172.21.15.86:0/22280154 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb264009450 con 0x7fb274077cd0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb272ffd640 1 --2- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb2741130d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb2640317f0 tx=0x7fb26402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb25b7fe640 1 -- 172.21.15.86:0/22280154 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb264032900 con 0x7fb274077cd0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb274113810 con 0x7fb274077cd0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb25b7fe640 1 -- 172.21.15.86:0/22280154 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb26403b910 con 0x7fb274077cd0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.349+0000 7fb25b7fe640 1 -- 172.21.15.86:0/22280154 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb26403a960 con 0x7fb274077cd0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.350+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb274113c80 con 0x7fb274077cd0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.350+0000 7fb25b7fe640 1 -- 172.21.15.86:0/22280154 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fb264039070 con 0x7fb274077cd0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.350+0000 7fb25b7fe640 1 --2- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb25c034810 0x7fb25c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.350+0000 7fb25b7fe640 1 -- 172.21.15.86:0/22280154 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb264066e80 con 0x7fb274077cd0 2024-06-06T08:22:16.221 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.350+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb274113c80 con 0x7fb274077cd0 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.351+0000 7fb2727fc640 1 --2- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb25c034810 0x7fb25c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.351+0000 7fb2727fc640 1 --2- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb25c034810 0x7fb25c036cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb268006fd0 tx=0x7fb268006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.355+0000 7fb25b7fe640 1 -- 172.21.15.86:0/22280154 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb274113c80 con 0x7fb274077cd0 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.536+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb274113c80 con 0x7fb25c034810 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.541+0000 7fb25b7fe640 1 -- 172.21.15.86:0/22280154 <== mgr.14118 v2:172.21.15.86:6800/933334455 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb274113c80 con 0x7fb25c034810 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb25c034810 msgr2=0x7fb25c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 --2- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb25c034810 0x7fb25c036cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb268006fd0 tx=0x7fb268006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 msgr2=0x7fb2741130d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 --2- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb2741130d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb2640317f0 tx=0x7fb26402c040 comp rx=0 tx=0).stop 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 shutdown_connections 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 --2- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7fb25c034810 0x7fb25c036cc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 --2- 172.21.15.86:0/22280154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb274077cd0 0x7fb2741130d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 >> 172.21.15.86:0/22280154 conn(0x7fb2740fd490 msgr2=0x7fb2740fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:16.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 shutdown_connections 2024-06-06T08:22:16.223 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:15.543+0000 7fb279343640 1 -- 172.21.15.86:0/22280154 wait complete. 2024-06-06T08:22:16.801 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:16 smithi086 ceph-mon[29114]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:16.801 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:16 smithi086 ceph-mon[29114]: Saving service alertmanager spec with placement count:1 2024-06-06T08:22:16.802 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:16 smithi086 ceph-mon[29114]: from='mgr.14118 172.21.15.86:0/1666556271' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:17.677 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:17 smithi086 ceph-mon[29114]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:22:18.311 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.501+0000 7f8285da3640 1 Processor -- start 2024-06-06T08:22:18.311 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.501+0000 7f8285da3640 1 -- start start 2024-06-06T08:22:18.311 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.502+0000 7f8285da3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f82801021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:18.311 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.502+0000 7f8285da3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8280102700 con 0x7f8280101db0 2024-06-06T08:22:18.311 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.502+0000 7f827f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f82801021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:18.311 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.502+0000 7f827f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f82801021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58390/0 (socket says 172.21.15.86:58390) 2024-06-06T08:22:18.311 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.502+0000 7f827f7fe640 1 -- 172.21.15.86:0/3352462896 learned_addr learned my addr 172.21.15.86:0/3352462896 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:18.312 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.502+0000 7f827f7fe640 1 -- 172.21.15.86:0/3352462896 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280102840 con 0x7f8280101db0 2024-06-06T08:22:18.312 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.503+0000 7f827f7fe640 1 --2- 172.21.15.86:0/3352462896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f82801021c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8270009840 tx=0x7f82700091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bcd4060b16bdcb server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:18.312 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.503+0000 7f827e7fc640 1 -- 172.21.15.86:0/3352462896 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8270032440 con 0x7f8280101db0 2024-06-06T08:22:18.315 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.503+0000 7f827e7fc640 1 -- 172.21.15.86:0/3352462896 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8270032a70 con 0x7f8280101db0 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.503+0000 7f827e7fc640 1 -- 172.21.15.86:0/3352462896 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8270031ad0 con 0x7f8280101db0 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.504+0000 7f8285da3640 1 -- 172.21.15.86:0/3352462896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 msgr2=0x7f82801021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.504+0000 7f8285da3640 1 --2- 172.21.15.86:0/3352462896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f82801021c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8270009840 tx=0x7f82700091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.504+0000 7f8285da3640 1 -- 172.21.15.86:0/3352462896 shutdown_connections 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.504+0000 7f8285da3640 1 --2- 172.21.15.86:0/3352462896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f82801021c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.504+0000 7f8285da3640 1 -- 172.21.15.86:0/3352462896 >> 172.21.15.86:0/3352462896 conn(0x7f82800fd5b0 msgr2=0x7f82800ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.504+0000 7f8285da3640 1 -- 172.21.15.86:0/3352462896 shutdown_connections 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.504+0000 7f8285da3640 1 -- 172.21.15.86:0/3352462896 wait complete. 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.504+0000 7f8285da3640 1 Processor -- start 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f8285da3640 1 -- start start 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f8285da3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f8280112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:18.316 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f8285da3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8280113330 con 0x7f8280101db0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f827f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f8280112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f827f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f8280112df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58392/0 (socket says 172.21.15.86:58392) 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f827f7fe640 1 -- 172.21.15.86:0/2888138598 learned_addr learned my addr 172.21.15.86:0/2888138598 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f827f7fe640 1 -- 172.21.15.86:0/2888138598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8270009450 con 0x7f8280101db0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f827f7fe640 1 --2- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f8280112df0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8270009810 tx=0x7f827002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f8267fff640 1 -- 172.21.15.86:0/2888138598 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82700328b0 con 0x7f8280101db0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f8267fff640 1 -- 172.21.15.86:0/2888138598 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f827003b910 con 0x7f8280101db0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8280113530 con 0x7f8280101db0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.505+0000 7f8267fff640 1 -- 172.21.15.86:0/2888138598 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f827003a980 con 0x7f8280101db0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.506+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8280113f80 con 0x7f8280101db0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.506+0000 7f8267fff640 1 -- 172.21.15.86:0/2888138598 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f8270039070 con 0x7f8280101db0 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.506+0000 7f8267fff640 1 --2- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8268034810 0x7f8268036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:18.317 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.507+0000 7f8267fff640 1 -- 172.21.15.86:0/2888138598 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8270066f30 con 0x7f8280101db0 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.507+0000 7f827effd640 1 --2- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8268034810 0x7f8268036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.507+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828005ead0 con 0x7f8280101db0 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.507+0000 7f827effd640 1 --2- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8268034810 0x7f8268036cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8274006fd0 tx=0x7f8274006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.511+0000 7f8267fff640 1 -- 172.21.15.86:0/2888138598 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8270037070 con 0x7f8280101db0 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.678+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f828004b480 con 0x7f8280101db0 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.681+0000 7f8267fff640 1 -- 172.21.15.86:0/2888138598 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8270030930 con 0x7f8280101db0 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8268034810 msgr2=0x7f8268036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 --2- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8268034810 0x7f8268036cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8274006fd0 tx=0x7f8274006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 msgr2=0x7f8280112df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 --2- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f8280112df0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8270009810 tx=0x7f827002c040 comp rx=0 tx=0).stop 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 shutdown_connections 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 --2- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f8268034810 0x7f8268036cc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:18.318 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 --2- 172.21.15.86:0/2888138598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8280101db0 0x7f8280112df0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:18.319 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 >> 172.21.15.86:0/2888138598 conn(0x7f82800fd5b0 msgr2=0x7f82800fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:18.319 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 shutdown_connections 2024-06-06T08:22:18.319 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:17.683+0000 7f8285da3640 1 -- 172.21.15.86:0/2888138598 wait complete. 2024-06-06T08:22:18.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:18 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2888138598' entity='client.admin' 2024-06-06T08:22:19.440 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.670+0000 7f40287e2640 1 Processor -- start 2024-06-06T08:22:19.440 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.670+0000 7f40287e2640 1 -- start start 2024-06-06T08:22:19.440 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.670+0000 7f40287e2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020076260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:19.440 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.671+0000 7f40287e2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4020078000 con 0x7f4020077bf0 2024-06-06T08:22:19.440 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.671+0000 7f402657e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020076260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:19.440 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.671+0000 7f402657e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020076260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58396/0 (socket says 172.21.15.86:58396) 2024-06-06T08:22:19.441 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.671+0000 7f402657e640 1 -- 172.21.15.86:0/3252005298 learned_addr learned my addr 172.21.15.86:0/3252005298 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:19.441 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.671+0000 7f402657e640 1 -- 172.21.15.86:0/3252005298 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020078140 con 0x7f4020077bf0 2024-06-06T08:22:19.441 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.672+0000 7f402657e640 1 --2- 172.21.15.86:0/3252005298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020076260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f401c009840 tx=0x7f401c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80175d42fb7c4ba7 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:19.441 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.672+0000 7f402557c640 1 -- 172.21.15.86:0/3252005298 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f401c032440 con 0x7f4020077bf0 2024-06-06T08:22:19.441 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.672+0000 7f402557c640 1 -- 172.21.15.86:0/3252005298 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f401c032a70 con 0x7f4020077bf0 2024-06-06T08:22:19.441 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.672+0000 7f402557c640 1 -- 172.21.15.86:0/3252005298 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f401c031ad0 con 0x7f4020077bf0 2024-06-06T08:22:19.441 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.672+0000 7f40287e2640 1 -- 172.21.15.86:0/3252005298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 msgr2=0x7f4020076260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:19.441 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.672+0000 7f40287e2640 1 --2- 172.21.15.86:0/3252005298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020076260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f401c009840 tx=0x7f401c0091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:19.441 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f40287e2640 1 -- 172.21.15.86:0/3252005298 shutdown_connections 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f40287e2640 1 --2- 172.21.15.86:0/3252005298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020076260 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f40287e2640 1 -- 172.21.15.86:0/3252005298 >> 172.21.15.86:0/3252005298 conn(0x7f40200fd5b0 msgr2=0x7f40200ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f40287e2640 1 -- 172.21.15.86:0/3252005298 shutdown_connections 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f40287e2640 1 -- 172.21.15.86:0/3252005298 wait complete. 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f40287e2640 1 Processor -- start 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f40287e2640 1 -- start start 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f40287e2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020112e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f40287e2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4020113370 con 0x7f4020077bf0 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f402657e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020112e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f402657e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020112e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58400/0 (socket says 172.21.15.86:58400) 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f402657e640 1 -- 172.21.15.86:0/3524375460 learned_addr learned my addr 172.21.15.86:0/3524375460 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.673+0000 7f402657e640 1 -- 172.21.15.86:0/3524375460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c009450 con 0x7f4020077bf0 2024-06-06T08:22:19.442 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.674+0000 7f402657e640 1 --2- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020112e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f401c0317f0 tx=0x7f401c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.674+0000 7f4016ffd640 1 -- 172.21.15.86:0/3524375460 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f401c032900 con 0x7f4020077bf0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.674+0000 7f4016ffd640 1 -- 172.21.15.86:0/3524375460 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f401c03b910 con 0x7f4020077bf0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.674+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4020113570 con 0x7f4020077bf0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.674+0000 7f4016ffd640 1 -- 172.21.15.86:0/3524375460 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f401c03a960 con 0x7f4020077bf0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.674+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4020113fc0 con 0x7f4020077bf0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.675+0000 7f4016ffd640 1 -- 172.21.15.86:0/3524375460 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f401c039070 con 0x7f4020077bf0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.675+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f402005ead0 con 0x7f4020077bf0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.675+0000 7f4016ffd640 1 --2- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f400c034860 0x7f400c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.675+0000 7f4016ffd640 1 -- 172.21.15.86:0/3524375460 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f401c066df0 con 0x7f4020077bf0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.675+0000 7f4025d7d640 1 --2- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f400c034860 0x7f400c036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.675+0000 7f4025d7d640 1 --2- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f400c034860 0x7f400c036d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4010006fd0 tx=0x7f4010006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.679+0000 7f4016ffd640 1 -- 172.21.15.86:0/3524375460 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f401c06b070 con 0x7f4020077bf0 2024-06-06T08:22:19.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.846+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f402004b480 con 0x7f4020077bf0 2024-06-06T08:22:19.444 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.849+0000 7f4016ffd640 1 -- 172.21.15.86:0/3524375460 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f401c0490c0 con 0x7f4020077bf0 2024-06-06T08:22:19.444 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f400c034860 msgr2=0x7f400c036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:19.444 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 --2- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f400c034860 0x7f400c036d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4010006fd0 tx=0x7f4010006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:19.444 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 msgr2=0x7f4020112e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:19.444 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 --2- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020112e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f401c0317f0 tx=0x7f401c02c040 comp rx=0 tx=0).stop 2024-06-06T08:22:19.444 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 shutdown_connections 2024-06-06T08:22:19.444 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 --2- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f400c034860 0x7f400c036d10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:19.444 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 --2- 172.21.15.86:0/3524375460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4020077bf0 0x7f4020112e30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:19.445 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 >> 172.21.15.86:0/3524375460 conn(0x7f40200fd5b0 msgr2=0x7f40200fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:19.445 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 shutdown_connections 2024-06-06T08:22:19.445 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:18.851+0000 7f40287e2640 1 -- 172.21.15.86:0/3524375460 wait complete. 2024-06-06T08:22:19.445 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling the dashboard module... 2024-06-06T08:22:20.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:19 smithi086 ceph-mon[29114]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:22:20.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:19 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3524375460' entity='client.admin' 2024-06-06T08:22:21.123 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:20 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2326536042' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-06T08:22:21.413 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.851+0000 7f755aa33640 1 Processor -- start 2024-06-06T08:22:21.414 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.851+0000 7f755aa33640 1 -- start start 2024-06-06T08:22:21.414 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.852+0000 7f755aa33640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:21.414 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.852+0000 7f755aa33640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75541015a0 con 0x7f7554100c50 2024-06-06T08:22:21.414 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.852+0000 7f7553fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:21.414 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.852+0000 7f7553fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58410/0 (socket says 172.21.15.86:58410) 2024-06-06T08:22:21.414 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.852+0000 7f7553fff640 1 -- 172.21.15.86:0/1967445498 learned_addr learned my addr 172.21.15.86:0/1967445498 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:21.414 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.852+0000 7f7553fff640 1 -- 172.21.15.86:0/1967445498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75541016e0 con 0x7f7554100c50 2024-06-06T08:22:21.414 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.853+0000 7f7553fff640 1 --2- 172.21.15.86:0/1967445498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554101060 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7544009840 tx=0x7f75440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58938875197ab246 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:21.415 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.853+0000 7f7552ffd640 1 -- 172.21.15.86:0/1967445498 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7544032440 con 0x7f7554100c50 2024-06-06T08:22:21.415 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.853+0000 7f7552ffd640 1 -- 172.21.15.86:0/1967445498 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7544032a70 con 0x7f7554100c50 2024-06-06T08:22:21.415 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.853+0000 7f7552ffd640 1 -- 172.21.15.86:0/1967445498 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7544031ad0 con 0x7f7554100c50 2024-06-06T08:22:21.415 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.853+0000 7f755aa33640 1 -- 172.21.15.86:0/1967445498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 msgr2=0x7f7554101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:21.415 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.853+0000 7f755aa33640 1 --2- 172.21.15.86:0/1967445498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554101060 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7544009840 tx=0x7f75440091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:21.415 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.853+0000 7f755aa33640 1 -- 172.21.15.86:0/1967445498 shutdown_connections 2024-06-06T08:22:21.415 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.853+0000 7f755aa33640 1 --2- 172.21.15.86:0/1967445498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554101060 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:21.416 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.853+0000 7f755aa33640 1 -- 172.21.15.86:0/1967445498 >> 172.21.15.86:0/1967445498 conn(0x7f75540fc430 msgr2=0x7f75540fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:21.416 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.854+0000 7f755aa33640 1 -- 172.21.15.86:0/1967445498 shutdown_connections 2024-06-06T08:22:21.416 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.854+0000 7f755aa33640 1 -- 172.21.15.86:0/1967445498 wait complete. 2024-06-06T08:22:21.416 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.854+0000 7f755aa33640 1 Processor -- start 2024-06-06T08:22:21.416 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.854+0000 7f755aa33640 1 -- start start 2024-06-06T08:22:21.416 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f755aa33640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554110bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:21.416 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f755aa33640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75541110f0 con 0x7f7554100c50 2024-06-06T08:22:21.416 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f7553fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554110bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:21.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f7553fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554110bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58420/0 (socket says 172.21.15.86:58420) 2024-06-06T08:22:21.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f7553fff640 1 -- 172.21.15.86:0/2326536042 learned_addr learned my addr 172.21.15.86:0/2326536042 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:21.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f7553fff640 1 -- 172.21.15.86:0/2326536042 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7544009450 con 0x7f7554100c50 2024-06-06T08:22:21.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f7553fff640 1 --2- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554110bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7544006a70 tx=0x7f754402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:21.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f7550ff9640 1 -- 172.21.15.86:0/2326536042 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7544031e90 con 0x7f7554100c50 2024-06-06T08:22:21.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75541112f0 con 0x7f7554100c50 2024-06-06T08:22:21.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f7550ff9640 1 -- 172.21.15.86:0/2326536042 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f754403b910 con 0x7f7554100c50 2024-06-06T08:22:21.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.855+0000 7f7550ff9640 1 -- 172.21.15.86:0/2326536042 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f754403a940 con 0x7f7554100c50 2024-06-06T08:22:21.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.856+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7554111760 con 0x7f7554100c50 2024-06-06T08:22:21.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.856+0000 7f7550ff9640 1 -- 172.21.15.86:0/2326536042 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f7544039070 con 0x7f7554100c50 2024-06-06T08:22:21.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.856+0000 7f7550ff9640 1 --2- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f753c034810 0x7f753c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:21.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.857+0000 7f7550ff9640 1 -- 172.21.15.86:0/2326536042 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7544067070 con 0x7f7554100c50 2024-06-06T08:22:21.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.857+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755405ead0 con 0x7f7554100c50 2024-06-06T08:22:21.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.857+0000 7f75537fe640 1 --2- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f753c034810 0x7f753c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:21.418 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.857+0000 7f75537fe640 1 --2- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f753c034810 0x7f753c036cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7548006fd0 tx=0x7f7548006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:21.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:19.861+0000 7f7550ff9640 1 -- 172.21.15.86:0/2326536042 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f754403f020 con 0x7f7554100c50 2024-06-06T08:22:21.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.068+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f755404b480 con 0x7f7554100c50 2024-06-06T08:22:21.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.853+0000 7f7550ff9640 1 -- 172.21.15.86:0/2326536042 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f75440412c0 con 0x7f7554100c50 2024-06-06T08:22:21.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.854+0000 7f7550ff9640 1 -- 172.21.15.86:0/2326536042 <== mon.0 v2:172.21.15.86:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f7544030340 con 0x7f7554100c50 2024-06-06T08:22:21.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f753c034810 msgr2=0x7f753c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:21.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 --2- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f753c034810 0x7f753c036cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7548006fd0 tx=0x7f7548006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:21.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 msgr2=0x7f7554110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:21.419 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 --2- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554110bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7544006a70 tx=0x7f754402c040 comp rx=0 tx=0).stop 2024-06-06T08:22:21.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 shutdown_connections 2024-06-06T08:22:21.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 --2- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f753c034810 0x7f753c036cc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:21.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 --2- 172.21.15.86:0/2326536042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7554100c50 0x7f7554110bb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:21.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 >> 172.21.15.86:0/2326536042 conn(0x7f75540fc430 msgr2=0x7f75540fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:21.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 shutdown_connections 2024-06-06T08:22:21.420 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:20.856+0000 7f755aa33640 1 -- 172.21.15.86:0/2326536042 wait complete. 2024-06-06T08:22:22.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:21 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2326536042' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-06T08:22:22.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:21 smithi086 ceph-mon[29114]: mgrmap e9: smithi086.mfkgze(active, since 24s) 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "active_name": "smithi086.mfkgze", 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.833+0000 7ff791507640 1 Processor -- start 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.833+0000 7ff791507640 1 -- start start 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.834+0000 7ff791507640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c104960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.834+0000 7ff791507640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c104ea0 con 0x7ff78c075bb0 2024-06-06T08:22:22.635 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.834+0000 7ff78affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.834+0000 7ff78affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c104960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58452/0 (socket says 172.21.15.86:58452) 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.834+0000 7ff78affd640 1 -- 172.21.15.86:0/1096470342 learned_addr learned my addr 172.21.15.86:0/1096470342 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.834+0000 7ff78affd640 1 -- 172.21.15.86:0/1096470342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c104fe0 con 0x7ff78c075bb0 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.835+0000 7ff78affd640 1 --2- 172.21.15.86:0/1096470342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c104960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009840 tx=0x7ff77c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf16bee48055251 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.835+0000 7ff789ffb640 1 -- 172.21.15.86:0/1096470342 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff77c032440 con 0x7ff78c075bb0 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.835+0000 7ff789ffb640 1 -- 172.21.15.86:0/1096470342 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff77c032a70 con 0x7ff78c075bb0 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.835+0000 7ff789ffb640 1 -- 172.21.15.86:0/1096470342 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff77c031ad0 con 0x7ff78c075bb0 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.835+0000 7ff791507640 1 -- 172.21.15.86:0/1096470342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 msgr2=0x7ff78c104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.835+0000 7ff791507640 1 --2- 172.21.15.86:0/1096470342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c104960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009840 tx=0x7ff77c0091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.835+0000 7ff791507640 1 -- 172.21.15.86:0/1096470342 shutdown_connections 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.836+0000 7ff791507640 1 --2- 172.21.15.86:0/1096470342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c104960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.836+0000 7ff791507640 1 -- 172.21.15.86:0/1096470342 >> 172.21.15.86:0/1096470342 conn(0x7ff78c0fc4b0 msgr2=0x7ff78c0fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:22.636 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.836+0000 7ff791507640 1 -- 172.21.15.86:0/1096470342 shutdown_connections 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.836+0000 7ff791507640 1 -- 172.21.15.86:0/1096470342 wait complete. 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.836+0000 7ff791507640 1 Processor -- start 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.836+0000 7ff791507640 1 -- start start 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.836+0000 7ff791507640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c110b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.836+0000 7ff791507640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c1110a0 con 0x7ff78c075bb0 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff78affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c110b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff78affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c110b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58458/0 (socket says 172.21.15.86:58458) 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff78affd640 1 -- 172.21.15.86:0/3292087349 learned_addr learned my addr 172.21.15.86:0/3292087349 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff78affd640 1 -- 172.21.15.86:0/3292087349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c009450 con 0x7ff78c075bb0 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff78affd640 1 --2- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c110b60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0317f0 tx=0x7ff77c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff7737fe640 1 -- 172.21.15.86:0/3292087349 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff77c032900 con 0x7ff78c075bb0 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff7737fe640 1 -- 172.21.15.86:0/3292087349 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff77c03b910 con 0x7ff78c075bb0 2024-06-06T08:22:22.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c1112a0 con 0x7ff78c075bb0 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff7737fe640 1 -- 172.21.15.86:0/3292087349 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff77c03a960 con 0x7ff78c075bb0 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.837+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c111710 con 0x7ff78c075bb0 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.838+0000 7ff7737fe640 1 -- 172.21.15.86:0/3292087349 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7ff77c039070 con 0x7ff78c075bb0 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.838+0000 7ff7737fe640 1 --2- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7ff7740348b0 0x7ff774036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.838+0000 7ff7737fe640 1 -- 172.21.15.86:0/3292087349 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff77c03b440 con 0x7ff78c075bb0 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.838+0000 7ff78a7fc640 1 -- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7ff7740348b0 msgr2=0x7ff774036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/933334455 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.838+0000 7ff78a7fc640 1 --2- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7ff7740348b0 0x7ff774036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.838+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78c05ead0 con 0x7ff78c075bb0 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:21.843+0000 7ff7737fe640 1 -- 172.21.15.86:0/3292087349 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff77c03f080 con 0x7ff78c075bb0 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.039+0000 7ff78a7fc640 1 -- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7ff7740348b0 msgr2=0x7ff774036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/933334455 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.039+0000 7ff78a7fc640 1 --2- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7ff7740348b0 0x7ff774036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T08:22:22.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.078+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff78c111f30 con 0x7ff78c075bb0 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.078+0000 7ff7737fe640 1 -- 172.21.15.86:0/3292087349 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7ff77c037070 con 0x7ff78c075bb0 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7ff7740348b0 msgr2=0x7ff774036d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 --2- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7ff7740348b0 0x7ff774036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 msgr2=0x7ff78c110b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 --2- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c110b60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0317f0 tx=0x7ff77c02c040 comp rx=0 tx=0).stop 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 shutdown_connections 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 --2- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7ff7740348b0 0x7ff774036d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 --2- 172.21.15.86:0/3292087349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff78c075bb0 0x7ff78c110b60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 >> 172.21.15.86:0/3292087349 conn(0x7ff78c0fc4b0 msgr2=0x7ff78c0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 shutdown_connections 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:22.080+0000 7ff791507640 1 -- 172.21.15.86:0/3292087349 wait complete. 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for the mgr to restart... 2024-06-06T08:22:22.639 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr epoch 9... 2024-06-06T08:22:23.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:22 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3292087349' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-06T08:22:25.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: Active manager daemon smithi086.mfkgze restarted 2024-06-06T08:22:25.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: Activating manager daemon smithi086.mfkgze 2024-06-06T08:22:25.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: osdmap e3: 0 total, 0 up, 0 in 2024-06-06T08:22:25.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: mgrmap e10: smithi086.mfkgze(active, starting, since 0.00301815s) 2024-06-06T08:22:25.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-06-06T08:22:25.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr metadata", "who": "smithi086.mfkgze", "id": "smithi086.mfkgze"}]: dispatch 2024-06-06T08:22:25.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T08:22:25.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T08:22:25.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T08:22:25.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: Manager daemon smithi086.mfkgze is now available 2024-06-06T08:22:25.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:22:25.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:22:25.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.mfkgze/mirror_snapshot_schedule"}]: dispatch 2024-06-06T08:22:25.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:25 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.mfkgze/trash_purge_schedule"}]: dispatch 2024-06-06T08:22:27.170 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-06T08:22:27.170 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-06T08:22:27.170 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-06T08:22:27.170 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-06T08:22:27.170 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.056+0000 7f2b2eb9c640 1 Processor -- start 2024-06-06T08:22:27.170 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.057+0000 7f2b2eb9c640 1 -- start start 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.057+0000 7f2b2eb9c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b281020a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.057+0000 7f2b2eb9c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b281025e0 con 0x7f2b28101c90 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.057+0000 7f2b2c938640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b281020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.057+0000 7f2b2c938640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b281020a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58470/0 (socket says 172.21.15.86:58470) 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.057+0000 7f2b2c938640 1 -- 172.21.15.86:0/1825166634 learned_addr learned my addr 172.21.15.86:0/1825166634 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.057+0000 7f2b2c938640 1 -- 172.21.15.86:0/1825166634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b28102720 con 0x7f2b28101c90 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.058+0000 7f2b2c938640 1 --2- 172.21.15.86:0/1825166634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b281020a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009840 tx=0x7f2b240091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d06e1ea4999613f server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.058+0000 7f2b1f7fe640 1 -- 172.21.15.86:0/1825166634 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b24032440 con 0x7f2b28101c90 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.058+0000 7f2b1f7fe640 1 -- 172.21.15.86:0/1825166634 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b24032a70 con 0x7f2b28101c90 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.058+0000 7f2b1f7fe640 1 -- 172.21.15.86:0/1825166634 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b24031ad0 con 0x7f2b28101c90 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.059+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/1825166634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 msgr2=0x7f2b281020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:27.171 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.059+0000 7f2b2eb9c640 1 --2- 172.21.15.86:0/1825166634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b281020a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009840 tx=0x7f2b240091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:27.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.059+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/1825166634 shutdown_connections 2024-06-06T08:22:27.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.059+0000 7f2b2eb9c640 1 --2- 172.21.15.86:0/1825166634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b281020a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:27.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.059+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/1825166634 >> 172.21.15.86:0/1825166634 conn(0x7f2b280fd490 msgr2=0x7f2b280ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:27.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.059+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/1825166634 shutdown_connections 2024-06-06T08:22:27.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.059+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/1825166634 wait complete. 2024-06-06T08:22:27.173 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.059+0000 7f2b2eb9c640 1 Processor -- start 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.060+0000 7f2b2eb9c640 1 -- start start 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.060+0000 7f2b2eb9c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b28077c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.060+0000 7f2b2eb9c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b28078150 con 0x7f2b28101c90 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.060+0000 7f2b2c938640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b28077c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.060+0000 7f2b2c938640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b28077c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58474/0 (socket says 172.21.15.86:58474) 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.060+0000 7f2b2c938640 1 -- 172.21.15.86:0/68684362 learned_addr learned my addr 172.21.15.86:0/68684362 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.060+0000 7f2b2c938640 1 -- 172.21.15.86:0/68684362 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24009450 con 0x7f2b28101c90 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.060+0000 7f2b2c938640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b28077c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b24006a70 tx=0x7f2b2402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.061+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b24031e90 con 0x7f2b28101c90 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.061+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b2403b910 con 0x7f2b28101c90 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.061+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/68684362 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b280761e0 con 0x7f2b28101c90 2024-06-06T08:22:27.174 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.061+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b2403a940 con 0x7f2b28101c90 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.061+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/68684362 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b28076650 con 0x7f2b28101c90 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.061+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f2b24039070 con 0x7f2b28101c90 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.062+0000 7f2b1d7fa640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 0x7f2b10036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.062+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 --> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b10037420 con 0x7f2b10034860 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.062+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2b24066fe0 con 0x7f2b28101c90 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.062+0000 7f2b1ffff640 1 -- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 msgr2=0x7f2b10036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/933334455 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.062+0000 7f2b1ffff640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 0x7f2b10036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.263+0000 7f2b1ffff640 1 -- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 msgr2=0x7f2b10036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/933334455 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.263+0000 7f2b1ffff640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 0x7f2b10036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.664+0000 7f2b1ffff640 1 -- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 msgr2=0x7f2b10036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/933334455 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:23.664+0000 7f2b1ffff640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 0x7f2b10036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-06T08:22:27.175 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:24.465+0000 7f2b1ffff640 1 -- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 msgr2=0x7f2b10036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/933334455 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:24.465+0000 7f2b1ffff640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 0x7f2b10036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:25.341+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mgrmap(e 10) v1 ==== 41726+0+0 (secure 0 0 0) 0x7f2b24066140 con 0x7f2b28101c90 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:25.341+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 msgr2=0x7f2b10036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:25.341+0000 7f2b1d7fa640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 0x7f2b10036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.342+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7f2b24042360 con 0x7f2b28101c90 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.342+0000 7f2b1d7fa640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f2b10038140 0x7f2b1003a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.342+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 --> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b10037420 con 0x7f2b10038140 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.343+0000 7f2b1ffff640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f2b10038140 0x7f2b1003a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.343+0000 7f2b1ffff640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f2b10038140 0x7f2b1003a520 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2b18003940 tx=0x7f2b18009350 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.344+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 <== mgr.14160 v2:172.21.15.86:6800/2470791934 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f2b10037420 con 0x7f2b10038140 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.349+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/68684362 --> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2b2804b480 con 0x7f2b10038140 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.349+0000 7f2b1d7fa640 1 -- 172.21.15.86:0/68684362 <== mgr.14160 v2:172.21.15.86:6800/2470791934 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f2b2804b480 con 0x7f2b10038140 2024-06-06T08:22:27.176 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f2b10038140 msgr2=0x7f2b1003a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f2b10038140 0x7f2b1003a520 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2b18003940 tx=0x7f2b18009350 comp rx=0 tx=0).stop 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 msgr2=0x7f2b28077c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b28077c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b24006a70 tx=0x7f2b2402c040 comp rx=0 tx=0).stop 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/68684362 shutdown_connections 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/933334455,v1:172.21.15.86:6801/933334455] conn(0x7f2b10034860 0x7f2b10036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f2b10038140 0x7f2b1003a520 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 --2- 172.21.15.86:0/68684362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b28101c90 0x7f2b28077c10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/68684362 >> 172.21.15.86:0/68684362 conn(0x7f2b280fd490 msgr2=0x7f2b280fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/68684362 shutdown_connections 2024-06-06T08:22:27.177 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:26.350+0000 7f2b2eb9c640 1 -- 172.21.15.86:0/68684362 wait complete. 2024-06-06T08:22:27.178 INFO:teuthology.orchestra.run.smithi086.stdout:mgr epoch 9 is available 2024-06-06T08:22:27.178 INFO:teuthology.orchestra.run.smithi086.stdout:Generating a dashboard self-signed certificate... 2024-06-06T08:22:27.583 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:27 smithi086 ceph-mon[29114]: [06/Jun/2024:08:22:26] ENGINE Bus STARTING 2024-06-06T08:22:27.583 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:27 smithi086 ceph-mon[29114]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-06T08:22:27.583 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:27 smithi086 ceph-mon[29114]: mgrmap e11: smithi086.mfkgze(active, since 1.00547s) 2024-06-06T08:22:27.583 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:27 smithi086 ceph-mon[29114]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-06T08:22:27.583 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:27 smithi086 ceph-mon[29114]: [06/Jun/2024:08:22:26] ENGINE Serving on https://172.21.15.86:7150 2024-06-06T08:22:27.583 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:27 smithi086 ceph-mon[29114]: [06/Jun/2024:08:22:26] ENGINE Bus STARTED 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:27 smithi086 ceph-mon[29114]: [06/Jun/2024:08:22:26] ENGINE Error in HTTPServer.serve 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Traceback (most recent call last): 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self._run(expiration_interval) 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: s = self.context.wrap_socket( 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-06T08:22:27.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: return self.sslsocket_class._create( 2024-06-06T08:22:27.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-06T08:22:27.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self.do_handshake() 2024-06-06T08:22:27.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-06T08:22:27.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self._sslobj.do_handshake() 2024-06-06T08:22:27.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-06T08:22:27.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:27 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:27.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:27 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-06T08:22:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.583+0000 7f16cdeb8640 1 Processor -- start 2024-06-06T08:22:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.584+0000 7f16cdeb8640 1 -- start start 2024-06-06T08:22:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.584+0000 7f16cdeb8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c81020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.584+0000 7f16cdeb8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8102600 con 0x7f16c8101cb0 2024-06-06T08:22:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.584+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c81020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.584+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c81020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60826/0 (socket says 172.21.15.86:60826) 2024-06-06T08:22:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.584+0000 7f16c77fe640 1 -- 172.21.15.86:0/2674583554 learned_addr learned my addr 172.21.15.86:0/2674583554 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.584+0000 7f16c77fe640 1 -- 172.21.15.86:0/2674583554 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c8102740 con 0x7f16c8101cb0 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.585+0000 7f16c77fe640 1 --2- 172.21.15.86:0/2674583554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c81020c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009840 tx=0x7f16b80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5185d68d0da824d server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.585+0000 7f16c6ffd640 1 -- 172.21.15.86:0/2674583554 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16b8032440 con 0x7f16c8101cb0 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.585+0000 7f16c6ffd640 1 -- 172.21.15.86:0/2674583554 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16b8032a70 con 0x7f16c8101cb0 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.585+0000 7f16c6ffd640 1 -- 172.21.15.86:0/2674583554 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16b8031ad0 con 0x7f16c8101cb0 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.585+0000 7f16cdeb8640 1 -- 172.21.15.86:0/2674583554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 msgr2=0x7f16c81020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.585+0000 7f16cdeb8640 1 --2- 172.21.15.86:0/2674583554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c81020c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009840 tx=0x7f16b80091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16cdeb8640 1 -- 172.21.15.86:0/2674583554 shutdown_connections 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16cdeb8640 1 --2- 172.21.15.86:0/2674583554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c81020c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16cdeb8640 1 -- 172.21.15.86:0/2674583554 >> 172.21.15.86:0/2674583554 conn(0x7f16c80fd490 msgr2=0x7f16c80ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:28.435 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16cdeb8640 1 -- 172.21.15.86:0/2674583554 shutdown_connections 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16cdeb8640 1 -- 172.21.15.86:0/2674583554 wait complete. 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16cdeb8640 1 Processor -- start 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16cdeb8640 1 -- start start 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16cdeb8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c8110ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16cdeb8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8111430 con 0x7f16c8101cb0 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c8110ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.586+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c8110ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60834/0 (socket says 172.21.15.86:60834) 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.587+0000 7f16c77fe640 1 -- 172.21.15.86:0/1650276328 learned_addr learned my addr 172.21.15.86:0/1650276328 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.587+0000 7f16c77fe640 1 -- 172.21.15.86:0/1650276328 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8009450 con 0x7f16c8101cb0 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.587+0000 7f16c77fe640 1 --2- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c8110ef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009810 tx=0x7f16b802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.587+0000 7f16c4ff9640 1 -- 172.21.15.86:0/1650276328 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16b80328b0 con 0x7f16c8101cb0 2024-06-06T08:22:28.436 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.587+0000 7f16c4ff9640 1 -- 172.21.15.86:0/1650276328 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16b803b910 con 0x7f16c8101cb0 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.587+0000 7f16c4ff9640 1 -- 172.21.15.86:0/1650276328 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16b803a980 con 0x7f16c8101cb0 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.587+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c8111630 con 0x7f16c8101cb0 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.587+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c81120e0 con 0x7f16c8101cb0 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.588+0000 7f16c4ff9640 1 -- 172.21.15.86:0/1650276328 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7f16b8039070 con 0x7f16c8101cb0 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.588+0000 7f16c4ff9640 1 --2- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f16b4034730 0x7f16b4036be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.588+0000 7f16c4ff9640 1 -- 172.21.15.86:0/1650276328 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f16b8066f20 con 0x7f16c8101cb0 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.588+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c805ead0 con 0x7f16c8101cb0 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.588+0000 7f16beffd640 1 --2- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f16b4034730 0x7f16b4036be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.589+0000 7f16beffd640 1 --2- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f16b4034730 0x7f16b4036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f16b0006fd0 tx=0x7f16b0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:28.437 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.593+0000 7f16c4ff9640 1 -- 172.21.15.86:0/1650276328 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f16b8037070 con 0x7f16c8101cb0 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.784+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 --> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f16c8103cf0 con 0x7f16b4034730 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.834+0000 7f16c4ff9640 1 -- 172.21.15.86:0/1650276328 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f16b804c350 con 0x7f16c8101cb0 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.844+0000 7f16c4ff9640 1 -- 172.21.15.86:0/1650276328 <== mgr.14160 v2:172.21.15.86:6800/2470791934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f16c8103cf0 con 0x7f16b4034730 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f16b4034730 msgr2=0x7f16b4036be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 --2- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f16b4034730 0x7f16b4036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f16b0006fd0 tx=0x7f16b0006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 msgr2=0x7f16c8110ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 --2- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c8110ef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009810 tx=0x7f16b802c040 comp rx=0 tx=0).stop 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 shutdown_connections 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 --2- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f16b4034730 0x7f16b4036be0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 --2- 172.21.15.86:0/1650276328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16c8101cb0 0x7f16c8110ef0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:28.438 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 >> 172.21.15.86:0/1650276328 conn(0x7f16c80fd490 msgr2=0x7f16c80fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:28.439 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 shutdown_connections 2024-06-06T08:22:28.439 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:27.846+0000 7f16cdeb8640 1 -- 172.21.15.86:0/1650276328 wait complete. 2024-06-06T08:22:28.439 INFO:teuthology.orchestra.run.smithi086.stdout:Creating initial admin user... 2024-06-06T08:22:28.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:28 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:28.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:28 smithi086 ceph-mon[29114]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:28.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:28 smithi086 ceph-mon[29114]: mgrmap e12: smithi086.mfkgze(active, since 2s) 2024-06-06T08:22:28.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:28 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:28.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:28 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:30.180 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ExF24p7UrntxvzirZlPYCucqBT1OXpCoB3acSwgnAr94rfDwfqyI6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1717662149, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-06T08:22:30.180 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.926+0000 7f8a2f77e640 1 Processor -- start 2024-06-06T08:22:30.181 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.927+0000 7f8a2f77e640 1 -- start start 2024-06-06T08:22:30.181 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.927+0000 7f8a2f77e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a281015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:30.181 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.927+0000 7f8a2f77e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28101b20 con 0x7f8a281011d0 2024-06-06T08:22:30.181 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.927+0000 7f8a2d51a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a281015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:30.181 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.927+0000 7f8a2d51a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a281015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60836/0 (socket says 172.21.15.86:60836) 2024-06-06T08:22:30.181 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.927+0000 7f8a2d51a640 1 -- 172.21.15.86:0/4177983419 learned_addr learned my addr 172.21.15.86:0/4177983419 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:30.182 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.927+0000 7f8a2d51a640 1 -- 172.21.15.86:0/4177983419 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a28101c60 con 0x7f8a281011d0 2024-06-06T08:22:30.182 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.928+0000 7f8a2d51a640 1 --2- 172.21.15.86:0/4177983419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a281015e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009840 tx=0x7f8a240091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aac8acaa0e773b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:30.182 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.928+0000 7f8a1ffff640 1 -- 172.21.15.86:0/4177983419 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a24032440 con 0x7f8a281011d0 2024-06-06T08:22:30.182 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.928+0000 7f8a1ffff640 1 -- 172.21.15.86:0/4177983419 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a24032a70 con 0x7f8a281011d0 2024-06-06T08:22:30.182 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.928+0000 7f8a1ffff640 1 -- 172.21.15.86:0/4177983419 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a24031ad0 con 0x7f8a281011d0 2024-06-06T08:22:30.182 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 -- 172.21.15.86:0/4177983419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 msgr2=0x7f8a281015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:30.183 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 --2- 172.21.15.86:0/4177983419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a281015e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009840 tx=0x7f8a240091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:30.183 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 -- 172.21.15.86:0/4177983419 shutdown_connections 2024-06-06T08:22:30.183 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 --2- 172.21.15.86:0/4177983419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a281015e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:30.183 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 -- 172.21.15.86:0/4177983419 >> 172.21.15.86:0/4177983419 conn(0x7f8a280fbc20 msgr2=0x7f8a280fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:30.183 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 -- 172.21.15.86:0/4177983419 shutdown_connections 2024-06-06T08:22:30.183 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 -- 172.21.15.86:0/4177983419 wait complete. 2024-06-06T08:22:30.184 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 Processor -- start 2024-06-06T08:22:30.184 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 -- start start 2024-06-06T08:22:30.184 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a2810e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:30.184 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.929+0000 7f8a2f77e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2810ee60 con 0x7f8a281011d0 2024-06-06T08:22:30.184 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a2d51a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a2810e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:30.184 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a2d51a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a2810e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60848/0 (socket says 172.21.15.86:60848) 2024-06-06T08:22:30.184 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a2d51a640 1 -- 172.21.15.86:0/750151155 learned_addr learned my addr 172.21.15.86:0/750151155 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:30.185 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a2d51a640 1 -- 172.21.15.86:0/750151155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24009450 con 0x7f8a281011d0 2024-06-06T08:22:30.185 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a2d51a640 1 --2- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a2810e920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009810 tx=0x7f8a2402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:30.185 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a1dffb640 1 -- 172.21.15.86:0/750151155 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a240328b0 con 0x7f8a281011d0 2024-06-06T08:22:30.185 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a1dffb640 1 -- 172.21.15.86:0/750151155 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a2403b910 con 0x7f8a281011d0 2024-06-06T08:22:30.185 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2810f060 con 0x7f8a281011d0 2024-06-06T08:22:30.185 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a1dffb640 1 -- 172.21.15.86:0/750151155 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a2403a980 con 0x7f8a281011d0 2024-06-06T08:22:30.185 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.930+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2810f4d0 con 0x7f8a281011d0 2024-06-06T08:22:30.186 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.931+0000 7f8a1dffb640 1 -- 172.21.15.86:0/750151155 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f8a24039070 con 0x7f8a281011d0 2024-06-06T08:22:30.186 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.931+0000 7f8a1dffb640 1 --2- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8a14036b00 0x7f8a14038fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:30.186 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.931+0000 7f8a1dffb640 1 -- 172.21.15.86:0/750151155 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8a24066f40 con 0x7f8a281011d0 2024-06-06T08:22:30.186 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.932+0000 7f8a2cd19640 1 --2- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8a14036b00 0x7f8a14038fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:30.186 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.932+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2805ead0 con 0x7f8a281011d0 2024-06-06T08:22:30.186 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.932+0000 7f8a2cd19640 1 --2- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8a14036b00 0x7f8a14038fb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006fd0 tx=0x7f8a18006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:30.186 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:28.936+0000 7f8a1dffb640 1 -- 172.21.15.86:0/750151155 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8a24037070 con 0x7f8a281011d0 2024-06-06T08:22:30.187 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.126+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 --> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8a2805e8c0 con 0x7f8a14036b00 2024-06-06T08:22:30.187 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.353+0000 7f8a1dffb640 1 -- 172.21.15.86:0/750151155 <== mgr.14160 v2:172.21.15.86:6800/2470791934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f8a2805e8c0 con 0x7f8a14036b00 2024-06-06T08:22:30.187 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8a14036b00 msgr2=0x7f8a14038fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:30.187 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 --2- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8a14036b00 0x7f8a14038fb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006fd0 tx=0x7f8a18006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:30.187 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 msgr2=0x7f8a2810e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:30.187 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 --2- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a2810e920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009810 tx=0x7f8a2402c040 comp rx=0 tx=0).stop 2024-06-06T08:22:30.187 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 shutdown_connections 2024-06-06T08:22:30.188 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 --2- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8a14036b00 0x7f8a14038fb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:30.188 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 --2- 172.21.15.86:0/750151155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a281011d0 0x7f8a2810e920 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:30.188 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 >> 172.21.15.86:0/750151155 conn(0x7f8a280fbc20 msgr2=0x7f8a281051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:30.188 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 shutdown_connections 2024-06-06T08:22:30.188 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:29.355+0000 7f8a2f77e640 1 -- 172.21.15.86:0/750151155 wait complete. 2024-06-06T08:22:30.188 INFO:teuthology.orchestra.run.smithi086.stdout:Fetching dashboard port number... 2024-06-06T08:22:30.423 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:30 smithi086 ceph-mon[29114]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:30.423 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:30 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:31.571 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 8443 2024-06-06T08:22:31.571 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.617+0000 7f041a38d640 1 Processor -- start 2024-06-06T08:22:31.571 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.617+0000 7f041a38d640 1 -- start start 2024-06-06T08:22:31.571 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.617+0000 7f041a38d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f0414076260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:31.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.617+0000 7f041a38d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414078140 con 0x7f0414077d30 2024-06-06T08:22:31.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.617+0000 7f0413fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f0414076260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:31.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.617+0000 7f0413fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f0414076260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60860/0 (socket says 172.21.15.86:60860) 2024-06-06T08:22:31.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.617+0000 7f0413fff640 1 -- 172.21.15.86:0/3067633423 learned_addr learned my addr 172.21.15.86:0/3067633423 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:31.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.617+0000 7f0413fff640 1 -- 172.21.15.86:0/3067633423 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04140767a0 con 0x7f0414077d30 2024-06-06T08:22:31.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.618+0000 7f0413fff640 1 --2- 172.21.15.86:0/3067633423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f0414076260 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0404009840 tx=0x7f04040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9c30da3029b6a server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:31.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.618+0000 7f0412ffd640 1 -- 172.21.15.86:0/3067633423 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0404032440 con 0x7f0414077d30 2024-06-06T08:22:31.572 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.618+0000 7f0412ffd640 1 -- 172.21.15.86:0/3067633423 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0404032a70 con 0x7f0414077d30 2024-06-06T08:22:31.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.618+0000 7f0412ffd640 1 -- 172.21.15.86:0/3067633423 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0404031ad0 con 0x7f0414077d30 2024-06-06T08:22:31.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.619+0000 7f041a38d640 1 -- 172.21.15.86:0/3067633423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 msgr2=0x7f0414076260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:31.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.619+0000 7f041a38d640 1 --2- 172.21.15.86:0/3067633423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f0414076260 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0404009840 tx=0x7f04040091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:31.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.619+0000 7f041a38d640 1 -- 172.21.15.86:0/3067633423 shutdown_connections 2024-06-06T08:22:31.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.619+0000 7f041a38d640 1 --2- 172.21.15.86:0/3067633423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f0414076260 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:31.573 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.619+0000 7f041a38d640 1 -- 172.21.15.86:0/3067633423 >> 172.21.15.86:0/3067633423 conn(0x7f04140fd490 msgr2=0x7f04140ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.619+0000 7f041a38d640 1 -- 172.21.15.86:0/3067633423 shutdown_connections 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.619+0000 7f041a38d640 1 -- 172.21.15.86:0/3067633423 wait complete. 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.619+0000 7f041a38d640 1 Processor -- start 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.619+0000 7f041a38d640 1 -- start start 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f041a38d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f04141130d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f041a38d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414113610 con 0x7f0414077d30 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f0413fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f04141130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f0413fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f04141130d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60868/0 (socket says 172.21.15.86:60868) 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f0413fff640 1 -- 172.21.15.86:0/3496291971 learned_addr learned my addr 172.21.15.86:0/3496291971 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f0413fff640 1 -- 172.21.15.86:0/3496291971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0404009450 con 0x7f0414077d30 2024-06-06T08:22:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f0413fff640 1 --2- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f04141130d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f04040317f0 tx=0x7f040402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:31.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f0410ff9640 1 -- 172.21.15.86:0/3496291971 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0404032900 con 0x7f0414077d30 2024-06-06T08:22:31.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0414113810 con 0x7f0414077d30 2024-06-06T08:22:31.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f0410ff9640 1 -- 172.21.15.86:0/3496291971 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f040403b910 con 0x7f0414077d30 2024-06-06T08:22:31.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f0410ff9640 1 -- 172.21.15.86:0/3496291971 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f040403a960 con 0x7f0414077d30 2024-06-06T08:22:31.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.620+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0414113c80 con 0x7f0414077d30 2024-06-06T08:22:31.575 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.621+0000 7f0410ff9640 1 -- 172.21.15.86:0/3496291971 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f0404039070 con 0x7f0414077d30 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.621+0000 7f0410ff9640 1 --2- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f03fc034860 0x7f03fc036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.621+0000 7f0410ff9640 1 -- 172.21.15.86:0/3496291971 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0404067100 con 0x7f0414077d30 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.621+0000 7f04137fe640 1 --2- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f03fc034860 0x7f03fc036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.621+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041405ead0 con 0x7f0414077d30 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.622+0000 7f04137fe640 1 --2- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f03fc034860 0x7f03fc036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0408006fd0 tx=0x7f0408006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.626+0000 7f0410ff9640 1 -- 172.21.15.86:0/3496291971 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0404037030 con 0x7f0414077d30 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.797+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f041404b480 con 0x7f0414077d30 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.798+0000 7f0410ff9640 1 -- 172.21.15.86:0/3496291971 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f040404c350 con 0x7f0414077d30 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.799+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f03fc034860 msgr2=0x7f03fc036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.799+0000 7f041a38d640 1 --2- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f03fc034860 0x7f03fc036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0408006fd0 tx=0x7f0408006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.799+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 msgr2=0x7f04141130d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.799+0000 7f041a38d640 1 --2- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f04141130d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f04040317f0 tx=0x7f040402c040 comp rx=0 tx=0).stop 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.800+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 shutdown_connections 2024-06-06T08:22:31.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.800+0000 7f041a38d640 1 --2- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f03fc034860 0x7f03fc036d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:31.577 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.800+0000 7f041a38d640 1 --2- 172.21.15.86:0/3496291971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0414077d30 0x7f04141130d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:31.577 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.800+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 >> 172.21.15.86:0/3496291971 conn(0x7f04140fd490 msgr2=0x7f04140fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:31.577 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.800+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 shutdown_connections 2024-06-06T08:22:31.577 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:30.800+0000 7f041a38d640 1 -- 172.21.15.86:0/3496291971 wait complete. 2024-06-06T08:22:31.577 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T08:22:31.578 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-06-06T08:22:31.585 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T08:22:31.586 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-06T08:22:31.586 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-06-06T08:22:31.586 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-06T08:22:31.586 INFO:teuthology.orchestra.run.smithi086.stdout:Ceph Dashboard is now available at: 2024-06-06T08:22:31.587 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:31.587 INFO:teuthology.orchestra.run.smithi086.stdout: URL: https://smithi086.front.sepia.ceph.com:8443/ 2024-06-06T08:22:31.587 INFO:teuthology.orchestra.run.smithi086.stdout: User: admin 2024-06-06T08:22:31.587 INFO:teuthology.orchestra.run.smithi086.stdout: Password: x39ca5oew5 2024-06-06T08:22:31.587 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:31.587 INFO:teuthology.orchestra.run.smithi086.stdout:Saving cluster configuration to /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config directory 2024-06-06T08:22:31.587 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling autotune for osd_memory_target 2024-06-06T08:22:31.696 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:31 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3496291971' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-06T08:22:32.726 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:32 smithi086 ceph-mon[29114]: mgrmap e13: smithi086.mfkgze(active, since 6s) 2024-06-06T08:22:32.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.951+0000 7f6a744dc640 1 Processor -- start 2024-06-06T08:22:32.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.951+0000 7f6a744dc640 1 -- start start 2024-06-06T08:22:32.734 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.951+0000 7f6a744dc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c076280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.951+0000 7f6a744dc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c078070 con 0x7f6a6c077c60 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.951+0000 7f6a72278640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c076280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.951+0000 7f6a72278640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c076280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60872/0 (socket says 172.21.15.86:60872) 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.951+0000 7f6a72278640 1 -- 172.21.15.86:0/1750026957 learned_addr learned my addr 172.21.15.86:0/1750026957 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.952+0000 7f6a72278640 1 -- 172.21.15.86:0/1750026957 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c0767c0 con 0x7f6a6c077c60 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.952+0000 7f6a72278640 1 --2- 172.21.15.86:0/1750026957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c076280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009840 tx=0x7f6a680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd8be8c762b6584 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a71276640 1 -- 172.21.15.86:0/1750026957 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a68032440 con 0x7f6a6c077c60 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a71276640 1 -- 172.21.15.86:0/1750026957 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a68032a70 con 0x7f6a6c077c60 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a71276640 1 -- 172.21.15.86:0/1750026957 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a68031ad0 con 0x7f6a6c077c60 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a744dc640 1 -- 172.21.15.86:0/1750026957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 msgr2=0x7f6a6c076280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a744dc640 1 --2- 172.21.15.86:0/1750026957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c076280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009840 tx=0x7f6a680091d0 comp rx=0 tx=0).stop 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a744dc640 1 -- 172.21.15.86:0/1750026957 shutdown_connections 2024-06-06T08:22:32.735 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a744dc640 1 --2- 172.21.15.86:0/1750026957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c076280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a744dc640 1 -- 172.21.15.86:0/1750026957 >> 172.21.15.86:0/1750026957 conn(0x7f6a6c0fd5b0 msgr2=0x7f6a6c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a744dc640 1 -- 172.21.15.86:0/1750026957 shutdown_connections 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.953+0000 7f6a744dc640 1 -- 172.21.15.86:0/1750026957 wait complete. 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.954+0000 7f6a744dc640 1 Processor -- start 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.954+0000 7f6a744dc640 1 -- start start 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.954+0000 7f6a744dc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c113100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.954+0000 7f6a744dc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c113640 con 0x7f6a6c077c60 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.954+0000 7f6a72278640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c113100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.954+0000 7f6a72278640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c113100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60888/0 (socket says 172.21.15.86:60888) 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.954+0000 7f6a72278640 1 -- 172.21.15.86:0/2321669551 learned_addr learned my addr 172.21.15.86:0/2321669551 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.954+0000 7f6a72278640 1 -- 172.21.15.86:0/2321669551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a68009450 con 0x7f6a6c077c60 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.954+0000 7f6a72278640 1 --2- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c113100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6a680317f0 tx=0x7f6a6802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:32.736 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.955+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2321669551 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a68032900 con 0x7f6a6c077c60 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.955+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2321669551 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a6803b910 con 0x7f6a6c077c60 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.955+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2321669551 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a6803a960 con 0x7f6a6c077c60 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.955+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6c113840 con 0x7f6a6c077c60 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.955+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6c113cb0 con 0x7f6a6c077c60 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.955+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2321669551 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f6a68039070 con 0x7f6a6c077c60 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.956+0000 7f6a62ffd640 1 --2- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6a58034900 0x7f6a58036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.956+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6c04b480 con 0x7f6a6c077c60 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.956+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2321669551 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6a68066e10 con 0x7f6a6c077c60 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.956+0000 7f6a71a77640 1 --2- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6a58034900 0x7f6a58036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.956+0000 7f6a71a77640 1 --2- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6a58034900 0x7f6a58036db0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c006fd0 tx=0x7f6a5c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:32.737 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:31.960+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2321669551 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6a68069030 con 0x7f6a6c077c60 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.128+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f6a6c063ed0 con 0x7f6a6c077c60 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.129+0000 7f6a62ffd640 1 -- 172.21.15.86:0/2321669551 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f6a6803f0c0 con 0x7f6a6c077c60 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6a58034900 msgr2=0x7f6a58036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 --2- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6a58034900 0x7f6a58036db0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c006fd0 tx=0x7f6a5c006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 msgr2=0x7f6a6c113100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 --2- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c113100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6a680317f0 tx=0x7f6a6802c040 comp rx=0 tx=0).stop 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 shutdown_connections 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 --2- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6a58034900 0x7f6a58036db0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 --2- 172.21.15.86:0/2321669551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c077c60 0x7f6a6c113100 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 >> 172.21.15.86:0/2321669551 conn(0x7f6a6c0fd5b0 msgr2=0x7f6a6c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:32.739 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 shutdown_connections 2024-06-06T08:22:32.739 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:32.130+0000 7f6a744dc640 1 -- 172.21.15.86:0/2321669551 wait complete. 2024-06-06T08:22:33.864 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.110+0000 7f476b9bb640 1 Processor -- start 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.111+0000 7f476b9bb640 1 -- start start 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.111+0000 7f476b9bb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f4764101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.111+0000 7f476b9bb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47641015a0 con 0x7f4764100c50 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.111+0000 7f4769757640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f4764101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.111+0000 7f4769757640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f4764101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60904/0 (socket says 172.21.15.86:60904) 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.111+0000 7f4769757640 1 -- 172.21.15.86:0/1276432184 learned_addr learned my addr 172.21.15.86:0/1276432184 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.111+0000 7f4769757640 1 -- 172.21.15.86:0/1276432184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47641016e0 con 0x7f4764100c50 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.112+0000 7f4769757640 1 --2- 172.21.15.86:0/1276432184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f4764101060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f476000baf0 tx=0x7f476000b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c06ace3846ef77 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.112+0000 7f475bfff640 1 -- 172.21.15.86:0/1276432184 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47600327a0 con 0x7f4764100c50 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.112+0000 7f475bfff640 1 -- 172.21.15.86:0/1276432184 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4760032dd0 con 0x7f4764100c50 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.112+0000 7f475bfff640 1 -- 172.21.15.86:0/1276432184 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4760039310 con 0x7f4764100c50 2024-06-06T08:22:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.113+0000 7f476b9bb640 1 -- 172.21.15.86:0/1276432184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 msgr2=0x7f4764101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.113+0000 7f476b9bb640 1 --2- 172.21.15.86:0/1276432184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f4764101060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f476000baf0 tx=0x7f476000b460 comp rx=0 tx=0).stop 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.113+0000 7f476b9bb640 1 -- 172.21.15.86:0/1276432184 shutdown_connections 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.113+0000 7f476b9bb640 1 --2- 172.21.15.86:0/1276432184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f4764101060 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.113+0000 7f476b9bb640 1 -- 172.21.15.86:0/1276432184 >> 172.21.15.86:0/1276432184 conn(0x7f47640fc430 msgr2=0x7f47640fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.113+0000 7f476b9bb640 1 -- 172.21.15.86:0/1276432184 shutdown_connections 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.113+0000 7f476b9bb640 1 -- 172.21.15.86:0/1276432184 wait complete. 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.113+0000 7f476b9bb640 1 Processor -- start 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.113+0000 7f476b9bb640 1 -- start start 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f476b9bb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f476410e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f476b9bb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476410ef00 con 0x7f4764100c50 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f4769757640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f476410e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:33.866 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f4769757640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f476410e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60914/0 (socket says 172.21.15.86:60914) 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f4769757640 1 -- 172.21.15.86:0/204819935 learned_addr learned my addr 172.21.15.86:0/204819935 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f4769757640 1 -- 172.21.15.86:0/204819935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476000b700 con 0x7f4764100c50 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f4769757640 1 --2- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f476410e9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f47600315c0 tx=0x7f476000b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f4759ffb640 1 -- 172.21.15.86:0/204819935 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47600329b0 con 0x7f4764100c50 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f4759ffb640 1 -- 172.21.15.86:0/204819935 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4760032b10 con 0x7f4764100c50 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476410f100 con 0x7f4764100c50 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.114+0000 7f4759ffb640 1 -- 172.21.15.86:0/204819935 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f476003b740 con 0x7f4764100c50 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.115+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476410f510 con 0x7f4764100c50 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.115+0000 7f4759ffb640 1 -- 172.21.15.86:0/204819935 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f476003b9b0 con 0x7f4764100c50 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.116+0000 7f4759ffb640 1 --2- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4750034900 0x7f4750036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.116+0000 7f4759ffb640 1 -- 172.21.15.86:0/204819935 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4760069b10 con 0x7f4764100c50 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.116+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476404b480 con 0x7f4764100c50 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.116+0000 7f4768f56640 1 --2- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4750034900 0x7f4750036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.116+0000 7f4768f56640 1 --2- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4750034900 0x7f4750036db0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4754006fd0 tx=0x7f4754006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.120+0000 7f4759ffb640 1 -- 172.21.15.86:0/204819935 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4760036070 con 0x7f4764100c50 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.371+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f476410fed0 con 0x7f4764100c50 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.373+0000 7f4759ffb640 1 -- 172.21.15.86:0/204819935 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f47600463f0 con 0x7f4764100c50 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.375+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4750034900 msgr2=0x7f4750036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.375+0000 7f476b9bb640 1 --2- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4750034900 0x7f4750036db0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4754006fd0 tx=0x7f4754006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.375+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 msgr2=0x7f476410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.375+0000 7f476b9bb640 1 --2- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f476410e9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f47600315c0 tx=0x7f476000b460 comp rx=0 tx=0).stop 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.376+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 shutdown_connections 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.376+0000 7f476b9bb640 1 --2- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4750034900 0x7f4750036db0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.376+0000 7f476b9bb640 1 --2- 172.21.15.86:0/204819935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4764100c50 0x7f476410e9c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.376+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 >> 172.21.15.86:0/204819935 conn(0x7f47640fc430 msgr2=0x7f47641052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:33.868 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.376+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 shutdown_connections 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr 2024-06-06T08:22:33.376+0000 7f476b9bb640 1 -- 172.21.15.86:0/204819935 wait complete. 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout:Or, if you are only running a single cluster on this host: 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: ceph telemetry on 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:33.869 INFO:teuthology.orchestra.run.smithi086.stdout:For more information see: 2024-06-06T08:22:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-06T08:22:33.870 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:33.870 INFO:teuthology.orchestra.run.smithi086.stdout:Bootstrap complete. 2024-06-06T08:22:33.897 INFO:tasks.cephadm:Fetching config... 2024-06-06T08:22:33.897 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:22:33.897 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-06T08:22:33.913 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-06T08:22:33.913 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:22:33.913 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-06T08:22:33.971 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-06T08:22:33.971 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:22:33.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/keyring of=/dev/stdout 2024-06-06T08:22:34.036 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-06T08:22:34.036 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:22:34.036 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-06T08:22:34.091 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-06T08:22:34.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDn5CxMUaXjagRRoHiZ059HRFHNprELnTIzIcV4CGpBWpHYfRl05AhOEB3LE587ycrtNy+4gM1SsT5xtPJoL67M+yCPFzRV+/uHnR9aihBVsvKr1YmpMutI5UnBZ4UhOta3ddxNrQ0+zEQ0Sm1H9VviUddbgg456WvhZ6VGWilkW+hWDGxHkc5czGTaPktux132rt0o0pknGJu10YonSFT+LONX7muhEgouoencErbsfBqRYsAFhPEiEXykBlz8XvbR5XmGsA0vGi2j3QrFsQljbyckIRWxFDNDqqbdzkWnt9FVg8XB+1VMc6UGxUcR/AX6WQYBfuv58I3Vttgdw5V0gHzH2JQJQA9z+IFGoYIoZhbBoNKXEtcyPJkCwNYCuKQJv6NYkKxTVSnO2KtNO0YXvHoY11/3z15HDHGs7ERhbutgdoKp/jIla0UQ9Je1gSjL0Zwr2V8CsmvR83YZ0l926w5inuPaIZ+otcSfcXgMf0G4BXzq5Bn4gSDW/hvpUfs= ceph-a8550dee-23dd-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-06T08:22:34.174 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDn5CxMUaXjagRRoHiZ059HRFHNprELnTIzIcV4CGpBWpHYfRl05AhOEB3LE587ycrtNy+4gM1SsT5xtPJoL67M+yCPFzRV+/uHnR9aihBVsvKr1YmpMutI5UnBZ4UhOta3ddxNrQ0+zEQ0Sm1H9VviUddbgg456WvhZ6VGWilkW+hWDGxHkc5czGTaPktux132rt0o0pknGJu10YonSFT+LONX7muhEgouoencErbsfBqRYsAFhPEiEXykBlz8XvbR5XmGsA0vGi2j3QrFsQljbyckIRWxFDNDqqbdzkWnt9FVg8XB+1VMc6UGxUcR/AX6WQYBfuv58I3Vttgdw5V0gHzH2JQJQA9z+IFGoYIoZhbBoNKXEtcyPJkCwNYCuKQJv6NYkKxTVSnO2KtNO0YXvHoY11/3z15HDHGs7ERhbutgdoKp/jIla0UQ9Je1gSjL0Zwr2V8CsmvR83YZ0l926w5inuPaIZ+otcSfcXgMf0G4BXzq5Bn4gSDW/hvpUfs= ceph-a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:22:34.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDn5CxMUaXjagRRoHiZ059HRFHNprELnTIzIcV4CGpBWpHYfRl05AhOEB3LE587ycrtNy+4gM1SsT5xtPJoL67M+yCPFzRV+/uHnR9aihBVsvKr1YmpMutI5UnBZ4UhOta3ddxNrQ0+zEQ0Sm1H9VviUddbgg456WvhZ6VGWilkW+hWDGxHkc5czGTaPktux132rt0o0pknGJu10YonSFT+LONX7muhEgouoencErbsfBqRYsAFhPEiEXykBlz8XvbR5XmGsA0vGi2j3QrFsQljbyckIRWxFDNDqqbdzkWnt9FVg8XB+1VMc6UGxUcR/AX6WQYBfuv58I3Vttgdw5V0gHzH2JQJQA9z+IFGoYIoZhbBoNKXEtcyPJkCwNYCuKQJv6NYkKxTVSnO2KtNO0YXvHoY11/3z15HDHGs7ERhbutgdoKp/jIla0UQ9Je1gSjL0Zwr2V8CsmvR83YZ0l926w5inuPaIZ+otcSfcXgMf0G4BXzq5Bn4gSDW/hvpUfs= ceph-a8550dee-23dd-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-06T08:22:34.227 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDn5CxMUaXjagRRoHiZ059HRFHNprELnTIzIcV4CGpBWpHYfRl05AhOEB3LE587ycrtNy+4gM1SsT5xtPJoL67M+yCPFzRV+/uHnR9aihBVsvKr1YmpMutI5UnBZ4UhOta3ddxNrQ0+zEQ0Sm1H9VviUddbgg456WvhZ6VGWilkW+hWDGxHkc5czGTaPktux132rt0o0pknGJu10YonSFT+LONX7muhEgouoencErbsfBqRYsAFhPEiEXykBlz8XvbR5XmGsA0vGi2j3QrFsQljbyckIRWxFDNDqqbdzkWnt9FVg8XB+1VMc6UGxUcR/AX6WQYBfuv58I3Vttgdw5V0gHzH2JQJQA9z+IFGoYIoZhbBoNKXEtcyPJkCwNYCuKQJv6NYkKxTVSnO2KtNO0YXvHoY11/3z15HDHGs7ERhbutgdoKp/jIla0UQ9Je1gSjL0Zwr2V8CsmvR83YZ0l926w5inuPaIZ+otcSfcXgMf0G4BXzq5Bn4gSDW/hvpUfs= ceph-a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:22:34.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-06T08:22:34.492 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:22:34.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:34 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/204819935' entity='client.admin' 2024-06-06T08:22:36.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.518+0000 7f4e7b121640 1 -- 172.21.15.86:0/450154286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 msgr2=0x7f4e740fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:36.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.518+0000 7f4e7b121640 1 --2- 172.21.15.86:0/450154286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 0x7f4e740fb450 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e68004850 tx=0x7f4e68009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:36.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.518+0000 7f4e7b121640 1 -- 172.21.15.86:0/450154286 shutdown_connections 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.518+0000 7f4e7b121640 1 --2- 172.21.15.86:0/450154286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 0x7f4e740fb450 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.518+0000 7f4e7b121640 1 -- 172.21.15.86:0/450154286 >> 172.21.15.86:0/450154286 conn(0x7f4e740f6820 msgr2=0x7f4e740f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.518+0000 7f4e7b121640 1 -- 172.21.15.86:0/450154286 shutdown_connections 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.518+0000 7f4e7b121640 1 -- 172.21.15.86:0/450154286 wait complete. 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.519+0000 7f4e7b121640 1 Processor -- start 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.519+0000 7f4e7b121640 1 -- start start 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.519+0000 7f4e7b121640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 0x7f4e74065230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.519+0000 7f4e7b121640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7410f480 con 0x7f4e740fb040 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.519+0000 7f4e78ebd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 0x7f4e74065230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.519+0000 7f4e78ebd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 0x7f4e74065230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60930/0 (socket says 172.21.15.86:60930) 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.519+0000 7f4e78ebd640 1 -- 172.21.15.86:0/3767656805 learned_addr learned my addr 172.21.15.86:0/3767656805 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.519+0000 7f4e78ebd640 1 -- 172.21.15.86:0/3767656805 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68005040 con 0x7f4e740fb040 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.520+0000 7f4e78ebd640 1 --2- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 0x7f4e74065230 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e68004820 tx=0x7f4e68009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.520+0000 7f4e717fa640 1 -- 172.21.15.86:0/3767656805 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e68032650 con 0x7f4e740fb040 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.520+0000 7f4e717fa640 1 -- 172.21.15.86:0/3767656805 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e68032c80 con 0x7f4e740fb040 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.520+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7410f5c0 con 0x7f4e740fb040 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.520+0000 7f4e717fa640 1 -- 172.21.15.86:0/3767656805 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e6803eda0 con 0x7f4e740fb040 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.520+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7410fa30 con 0x7f4e740fb040 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.521+0000 7f4e717fa640 1 -- 172.21.15.86:0/3767656805 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f4e68030020 con 0x7f4e740fb040 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.521+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e741015b0 con 0x7f4e740fb040 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.521+0000 7f4e717fa640 1 --2- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4e64034900 0x7f4e64036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.521+0000 7f4e717fa640 1 -- 172.21.15.86:0/3767656805 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4e68066910 con 0x7f4e740fb040 2024-06-06T08:22:36.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.521+0000 7f4e73fff640 1 --2- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4e64034900 0x7f4e64036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:36.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.522+0000 7f4e73fff640 1 --2- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4e64034900 0x7f4e64036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e60006fd0 tx=0x7f4e60006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:36.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.526+0000 7f4e717fa640 1 -- 172.21.15.86:0/3767656805 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4e68033420 con 0x7f4e740fb040 2024-06-06T08:22:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.702+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4e7404b500 con 0x7f4e740fb040 2024-06-06T08:22:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.705+0000 7f4e717fa640 1 -- 172.21.15.86:0/3767656805 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4e7404b500 con 0x7f4e740fb040 2024-06-06T08:22:36.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4e64034900 msgr2=0x7f4e64036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:36.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 --2- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4e64034900 0x7f4e64036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e60006fd0 tx=0x7f4e60006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:36.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 msgr2=0x7f4e74065230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:36.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 --2- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 0x7f4e74065230 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e68004820 tx=0x7f4e68009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:36.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 shutdown_connections 2024-06-06T08:22:36.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 --2- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f4e64034900 0x7f4e64036db0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:36.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 --2- 172.21.15.86:0/3767656805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e740fb040 0x7f4e74065230 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:36.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 >> 172.21.15.86:0/3767656805 conn(0x7f4e740f6820 msgr2=0x7f4e741021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:36.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 shutdown_connections 2024-06-06T08:22:36.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:36.708+0000 7f4e7b121640 1 -- 172.21.15.86:0/3767656805 wait complete. 2024-06-06T08:22:37.331 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:37 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:37.331 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:37 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:37.331 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:37 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:22:37.332 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:37 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:37.332 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:37 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T08:22:37.332 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:37 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-06T08:22:37.332 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:37 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:22:37.332 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:37 smithi086 ceph-mon[29114]: Deploying daemon crash.smithi086 on smithi086 2024-06-06T08:22:37.332 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:37 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3767656805' entity='client.admin' 2024-06-06T08:22:37.440 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-06T08:22:37.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-06T08:22:38.107 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:22:40.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.034+0000 7f72086e7640 1 -- 172.21.15.86:0/2620646268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 msgr2=0x7f72000fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.034+0000 7f72086e7640 1 --2- 172.21.15.86:0/2620646268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 0x7f72000fb450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f71fc004850 tx=0x7f71fc009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.034+0000 7f72086e7640 1 -- 172.21.15.86:0/2620646268 shutdown_connections 2024-06-06T08:22:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.034+0000 7f72086e7640 1 --2- 172.21.15.86:0/2620646268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 0x7f72000fb450 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.034+0000 7f72086e7640 1 -- 172.21.15.86:0/2620646268 >> 172.21.15.86:0/2620646268 conn(0x7f72000f6820 msgr2=0x7f72000f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.034+0000 7f72086e7640 1 -- 172.21.15.86:0/2620646268 shutdown_connections 2024-06-06T08:22:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.034+0000 7f72086e7640 1 -- 172.21.15.86:0/2620646268 wait complete. 2024-06-06T08:22:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.034+0000 7f72086e7640 1 Processor -- start 2024-06-06T08:22:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.034+0000 7f72086e7640 1 -- start start 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f72086e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 0x7f7200065230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f72086e7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720010f480 con 0x7f72000fb040 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f7206483640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 0x7f7200065230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f7206483640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 0x7f7200065230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53656/0 (socket says 172.21.15.86:53656) 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f7206483640 1 -- 172.21.15.86:0/2914718498 learned_addr learned my addr 172.21.15.86:0/2914718498 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f7206483640 1 -- 172.21.15.86:0/2914718498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc005040 con 0x7f72000fb040 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f7206483640 1 --2- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 0x7f7200065230 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0098d0 tx=0x7f71fc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f71f6ffd640 1 -- 172.21.15.86:0/2914718498 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71fc032650 con 0x7f72000fb040 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f71f6ffd640 1 -- 172.21.15.86:0/2914718498 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71fc032c80 con 0x7f72000fb040 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f71f6ffd640 1 -- 172.21.15.86:0/2914718498 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71fc033c10 con 0x7f72000fb040 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720010f5c0 con 0x7f72000fb040 2024-06-06T08:22:40.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.035+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720010f9d0 con 0x7f72000fb040 2024-06-06T08:22:40.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.036+0000 7f71f6ffd640 1 -- 172.21.15.86:0/2914718498 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f71fc030020 con 0x7f72000fb040 2024-06-06T08:22:40.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.036+0000 7f71f6ffd640 1 --2- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f71ec034900 0x7f71ec036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:40.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.036+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720005ead0 con 0x7f72000fb040 2024-06-06T08:22:40.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.036+0000 7f71f6ffd640 1 -- 172.21.15.86:0/2914718498 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f71fc0667f0 con 0x7f72000fb040 2024-06-06T08:22:40.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.036+0000 7f7205c82640 1 --2- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f71ec034900 0x7f71ec036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:40.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.037+0000 7f7205c82640 1 --2- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f71ec034900 0x7f71ec036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f71f0006fd0 tx=0x7f71f0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:40.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.041+0000 7f71f6ffd640 1 -- 172.21.15.86:0/2914718498 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f71fc0327b0 con 0x7f72000fb040 2024-06-06T08:22:40.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.216+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 --> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7200100ef0 con 0x7f71ec034900 2024-06-06T08:22:40.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.220+0000 7f71f6ffd640 1 -- 172.21.15.86:0/2914718498 <== mgr.14160 v2:172.21.15.86:6800/2470791934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7200100ef0 con 0x7f71ec034900 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.221+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f71ec034900 msgr2=0x7f71ec036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.221+0000 7f72086e7640 1 --2- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f71ec034900 0x7f71ec036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f71f0006fd0 tx=0x7f71f0006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.221+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 msgr2=0x7f7200065230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.221+0000 7f72086e7640 1 --2- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 0x7f7200065230 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0098d0 tx=0x7f71fc009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.221+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 shutdown_connections 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.221+0000 7f72086e7640 1 --2- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f71ec034900 0x7f71ec036db0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.221+0000 7f72086e7640 1 --2- 172.21.15.86:0/2914718498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72000fb040 0x7f7200065230 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.221+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 >> 172.21.15.86:0/2914718498 conn(0x7f72000f6820 msgr2=0x7f72001021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.222+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 shutdown_connections 2024-06-06T08:22:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:40.222+0000 7f72086e7640 1 -- 172.21.15.86:0/2914718498 wait complete. 2024-06-06T08:22:41.188 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi154 2024-06-06T08:22:41.189 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:22:41.189 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.conf 2024-06-06T08:22:41.208 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:22:41.208 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-06T08:22:41.266 INFO:tasks.cephadm:Adding host smithi154 to orchestrator... 2024-06-06T08:22:41.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch host add smithi154 2024-06-06T08:22:41.525 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:41 smithi086 ceph-mon[29114]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:41.525 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:41 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:41.762 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:22:42.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:42 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:42.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:42 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:42.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:42 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:42.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:42 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:42.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:42 smithi086 ceph-mon[29114]: Deploying daemon node-exporter.smithi086 on smithi086 2024-06-06T08:22:42.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:42 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:42.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:42 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:42.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:42 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:43.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.516+0000 7f01fffca640 1 -- 172.21.15.86:0/3918242259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 msgr2=0x7f01f80fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.516+0000 7f01fffca640 1 --2- 172.21.15.86:0/3918242259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 0x7f01f80fc360 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01f4004850 tx=0x7f01f4009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.516+0000 7f01fffca640 1 -- 172.21.15.86:0/3918242259 shutdown_connections 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.516+0000 7f01fffca640 1 --2- 172.21.15.86:0/3918242259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 0x7f01f80fc360 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.516+0000 7f01fffca640 1 -- 172.21.15.86:0/3918242259 >> 172.21.15.86:0/3918242259 conn(0x7f01f80f7710 msgr2=0x7f01f80f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.516+0000 7f01fffca640 1 -- 172.21.15.86:0/3918242259 shutdown_connections 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.516+0000 7f01fffca640 1 -- 172.21.15.86:0/3918242259 wait complete. 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.517+0000 7f01fffca640 1 Processor -- start 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.517+0000 7f01fffca640 1 -- start start 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.517+0000 7f01fffca640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 0x7f01f810c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.517+0000 7f01fffca640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f810cac0 con 0x7f01f80fbf50 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.517+0000 7f01fdd66640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 0x7f01f810c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.517+0000 7f01fdd66640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 0x7f01f810c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53722/0 (socket says 172.21.15.86:53722) 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.517+0000 7f01fdd66640 1 -- 172.21.15.86:0/463448447 learned_addr learned my addr 172.21.15.86:0/463448447 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.518+0000 7f01fdd66640 1 -- 172.21.15.86:0/463448447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f4005040 con 0x7f01f80fbf50 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.518+0000 7f01fdd66640 1 --2- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 0x7f01f810c580 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f01f4004820 tx=0x7f01f4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.518+0000 7f01ee7fc640 1 -- 172.21.15.86:0/463448447 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01f4032650 con 0x7f01f80fbf50 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.518+0000 7f01ee7fc640 1 -- 172.21.15.86:0/463448447 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01f4032c80 con 0x7f01f80fbf50 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.518+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f810ccc0 con 0x7f01f80fbf50 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.518+0000 7f01ee7fc640 1 -- 172.21.15.86:0/463448447 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01f403eda0 con 0x7f01f80fbf50 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.518+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f810d850 con 0x7f01f80fbf50 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.519+0000 7f01ee7fc640 1 -- 172.21.15.86:0/463448447 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f01f4030020 con 0x7f01f80fbf50 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.519+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f81024c0 con 0x7f01f80fbf50 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.519+0000 7f01ee7fc640 1 --2- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f01e4034900 0x7f01e4036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:43.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.519+0000 7f01ee7fc640 1 -- 172.21.15.86:0/463448447 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f01f4066930 con 0x7f01f80fbf50 2024-06-06T08:22:43.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.519+0000 7f01fd565640 1 --2- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f01e4034900 0x7f01e4036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:43.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.520+0000 7f01fd565640 1 --2- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f01e4034900 0x7f01e4036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f01e8006fd0 tx=0x7f01e8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:43.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.524+0000 7f01ee7fc640 1 -- 172.21.15.86:0/463448447 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f01f4033420 con 0x7f01f80fbf50 2024-06-06T08:22:43.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:43.708+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 --> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}) v1 -- 0x7f01f810da90 con 0x7f01e4034900 2024-06-06T08:22:43.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:43 smithi086 ceph-mon[29114]: from='client.14186 -' entity='client.crash.smithi086' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:43.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:43 smithi086 ceph-mon[29114]: from='client.14188 -' entity='client.crash.smithi086' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:43.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:43 smithi086 ceph-mon[29114]: from='client.14190 -' entity='client.crash.smithi086' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:45.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:44 smithi086 ceph-mon[29114]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:22:46.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:45 smithi086 ceph-mon[29114]: Deploying cephadm binary to smithi154 2024-06-06T08:22:46.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:45 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:46.630 INFO:teuthology.orchestra.run.smithi086.stdout:Added host 'smithi154' with addr '172.21.15.154' 2024-06-06T08:22:46.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.630+0000 7f01ee7fc640 1 -- 172.21.15.86:0/463448447 <== mgr.14160 v2:172.21.15.86:6800/2470791934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f01f810da90 con 0x7f01e4034900 2024-06-06T08:22:46.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f01e4034900 msgr2=0x7f01e4036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:46.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 --2- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f01e4034900 0x7f01e4036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f01e8006fd0 tx=0x7f01e8006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:46.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 msgr2=0x7f01f810c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:46.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 --2- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 0x7f01f810c580 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f01f4004820 tx=0x7f01f4009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:46.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 shutdown_connections 2024-06-06T08:22:46.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 --2- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f01e4034900 0x7f01e4036db0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:46.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 --2- 172.21.15.86:0/463448447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01f80fbf50 0x7f01f810c580 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:46.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 >> 172.21.15.86:0/463448447 conn(0x7f01f80f7710 msgr2=0x7f01f81030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:46.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 shutdown_connections 2024-06-06T08:22:46.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:46.632+0000 7f01fffca640 1 -- 172.21.15.86:0/463448447 wait complete. 2024-06-06T08:22:47.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-06T08:22:47.498 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:22:47.615 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:47 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:47.616 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:47 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:47.616 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:47 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:47.616 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:47 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:47.616 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:47 smithi086 ceph-mon[29114]: Deploying daemon alertmanager.smithi086 on smithi086 2024-06-06T08:22:47.616 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:47 smithi086 ceph-mon[29114]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:22:47.616 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:47 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:47.616 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:47 smithi086 ceph-mon[29114]: Added host smithi154 2024-06-06T08:22:48.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:48 smithi086 ceph-mon[29114]: Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-06T08:22:49.578 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:49 smithi086 ceph-mon[29114]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:22:49.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.661+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/3498975667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 msgr2=0x7fd5a00feba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:49.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.661+0000 7fd5a5ed0640 1 --2- 172.21.15.86:0/3498975667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 0x7fd5a00feba0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd590004850 tx=0x7fd590009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:49.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.661+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/3498975667 shutdown_connections 2024-06-06T08:22:49.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.661+0000 7fd5a5ed0640 1 --2- 172.21.15.86:0/3498975667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 0x7fd5a00feba0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.661+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/3498975667 >> 172.21.15.86:0/3498975667 conn(0x7fd5a00f65f0 msgr2=0x7fd5a00f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.662+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/3498975667 shutdown_connections 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.662+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/3498975667 wait complete. 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.662+0000 7fd5a5ed0640 1 Processor -- start 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.662+0000 7fd5a5ed0640 1 -- start start 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.662+0000 7fd5a5ed0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 0x7fd5a00fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.662+0000 7fd5a5ed0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00fca10 con 0x7fd5a0065ea0 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd59f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 0x7fd5a00fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd59f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 0x7fd5a00fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55280/0 (socket says 172.21.15.86:55280) 2024-06-06T08:22:49.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd59f7fe640 1 -- 172.21.15.86:0/271508800 learned_addr learned my addr 172.21.15.86:0/271508800 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd59f7fe640 1 -- 172.21.15.86:0/271508800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd590005040 con 0x7fd5a0065ea0 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd59f7fe640 1 --2- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 0x7fd5a00fc4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd590003080 tx=0x7fd590009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd587fff640 1 -- 172.21.15.86:0/271508800 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5900326c0 con 0x7fd5a0065ea0 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd587fff640 1 -- 172.21.15.86:0/271508800 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd590032cf0 con 0x7fd5a0065ea0 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a00faaa0 con 0x7fd5a0065ea0 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd587fff640 1 -- 172.21.15.86:0/271508800 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd590033db0 con 0x7fd5a0065ea0 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.663+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a00faf70 con 0x7fd5a0065ea0 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.664+0000 7fd587fff640 1 -- 172.21.15.86:0/271508800 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fd590030020 con 0x7fd5a0065ea0 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.664+0000 7fd587fff640 1 --2- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fd588034900 0x7fd588036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.664+0000 7fd587fff640 1 -- 172.21.15.86:0/271508800 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd590067790 con 0x7fd5a0065ea0 2024-06-06T08:22:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.665+0000 7fd59effd640 1 --2- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fd588034900 0x7fd588036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:49.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.665+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a005ead0 con 0x7fd5a0065ea0 2024-06-06T08:22:49.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.665+0000 7fd59effd640 1 --2- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fd588034900 0x7fd588036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd594006fd0 tx=0x7fd594006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:49.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.670+0000 7fd587fff640 1 -- 172.21.15.86:0/271508800 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd59003c380 con 0x7fd5a0065ea0 2024-06-06T08:22:49.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.841+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 --> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5a00fb1b0 con 0x7fd588034900 2024-06-06T08:22:49.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.842+0000 7fd587fff640 1 -- 172.21.15.86:0/271508800 <== mgr.14160 v2:172.21.15.86:6800/2470791934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fd5a00fb1b0 con 0x7fd588034900 2024-06-06T08:22:49.842 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:22:49.842 INFO:teuthology.orchestra.run.smithi086.stdout:[{"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}] 2024-06-06T08:22:49.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fd588034900 msgr2=0x7fd588036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:49.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 --2- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fd588034900 0x7fd588036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd594006fd0 tx=0x7fd594006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:49.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 msgr2=0x7fd5a00fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:49.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 --2- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 0x7fd5a00fc4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd590003080 tx=0x7fd590009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:49.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 shutdown_connections 2024-06-06T08:22:49.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 --2- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fd588034900 0x7fd588036db0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:49.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 --2- 172.21.15.86:0/271508800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5a0065ea0 0x7fd5a00fc4d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:49.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 >> 172.21.15.86:0/271508800 conn(0x7fd5a00f65f0 msgr2=0x7fd5a01030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:49.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 shutdown_connections 2024-06-06T08:22:49.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:49.844+0000 7fd5a5ed0640 1 -- 172.21.15.86:0/271508800 wait complete. 2024-06-06T08:22:50.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:50 smithi086 ceph-mon[29114]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T08:22:51.497 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-06T08:22:51.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-06-06T08:22:51.521 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:51 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:51.521 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:51 smithi086 ceph-mon[29114]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:22:52.649 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:22:54.020 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:53 smithi086 ceph-mon[29114]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:22:55.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.798+0000 7fad7adfa640 1 -- 172.21.15.86:0/3457739316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 msgr2=0x7fad740fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.798+0000 7fad7adfa640 1 --2- 172.21.15.86:0/3457739316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 0x7fad740fc320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fad68004850 tx=0x7fad68009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.799+0000 7fad7adfa640 1 -- 172.21.15.86:0/3457739316 shutdown_connections 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.799+0000 7fad7adfa640 1 --2- 172.21.15.86:0/3457739316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 0x7fad740fc320 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.799+0000 7fad7adfa640 1 -- 172.21.15.86:0/3457739316 >> 172.21.15.86:0/3457739316 conn(0x7fad740f76f0 msgr2=0x7fad740f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.799+0000 7fad7adfa640 1 -- 172.21.15.86:0/3457739316 shutdown_connections 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.799+0000 7fad7adfa640 1 -- 172.21.15.86:0/3457739316 wait complete. 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.799+0000 7fad7adfa640 1 Processor -- start 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.799+0000 7fad7adfa640 1 -- start start 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.799+0000 7fad7adfa640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 0x7fad740711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.799+0000 7fad7adfa640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad74071730 con 0x7fad740fbf10 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad78b96640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 0x7fad740711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad78b96640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 0x7fad740711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55310/0 (socket says 172.21.15.86:55310) 2024-06-06T08:22:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad78b96640 1 -- 172.21.15.86:0/3004290713 learned_addr learned my addr 172.21.15.86:0/3004290713 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:22:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad78b96640 1 -- 172.21.15.86:0/3004290713 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad68005040 con 0x7fad740fbf10 2024-06-06T08:22:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad78b96640 1 --2- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 0x7fad740711f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fad68004820 tx=0x7fad68009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad717fa640 1 -- 172.21.15.86:0/3004290713 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad68032650 con 0x7fad740fbf10 2024-06-06T08:22:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad717fa640 1 -- 172.21.15.86:0/3004290713 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad68032c80 con 0x7fad740fbf10 2024-06-06T08:22:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7406f7c0 con 0x7fad740fbf10 2024-06-06T08:22:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad717fa640 1 -- 172.21.15.86:0/3004290713 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad6803eda0 con 0x7fad740fbf10 2024-06-06T08:22:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.800+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7406fc90 con 0x7fad740fbf10 2024-06-06T08:22:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.801+0000 7fad717fa640 1 -- 172.21.15.86:0/3004290713 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fad68030020 con 0x7fad740fbf10 2024-06-06T08:22:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.801+0000 7fad717fa640 1 --2- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fad64034900 0x7fad64036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.801+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7405ead0 con 0x7fad740fbf10 2024-06-06T08:22:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.801+0000 7fad717fa640 1 -- 172.21.15.86:0/3004290713 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fad68066910 con 0x7fad740fbf10 2024-06-06T08:22:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.801+0000 7fad73fff640 1 --2- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fad64034900 0x7fad64036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:55.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.802+0000 7fad73fff640 1 --2- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fad64034900 0x7fad64036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fad60006fd0 tx=0x7fad60006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:55.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.806+0000 7fad717fa640 1 -- 172.21.15.86:0/3004290713 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fad6803c920 con 0x7fad740fbf10 2024-06-06T08:22:55.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:55.984+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fad7404b480 con 0x7fad740fbf10 2024-06-06T08:22:55.984 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:55 smithi086 ceph-mon[29114]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:22:56.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.645+0000 7fad717fa640 1 -- 172.21.15.86:0/3004290713 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fad6803c330 con 0x7fad740fbf10 2024-06-06T08:22:56.646 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2024-06-06T08:22:56.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.647+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fad64034900 msgr2=0x7fad64036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:56.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.647+0000 7fad7adfa640 1 --2- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fad64034900 0x7fad64036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fad60006fd0 tx=0x7fad60006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:56.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.648+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 msgr2=0x7fad740711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:56.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.648+0000 7fad7adfa640 1 --2- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 0x7fad740711f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fad68004820 tx=0x7fad68009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:56.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.648+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 shutdown_connections 2024-06-06T08:22:56.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.648+0000 7fad7adfa640 1 --2- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fad64034900 0x7fad64036db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:56.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.648+0000 7fad7adfa640 1 --2- 172.21.15.86:0/3004290713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad740fbf10 0x7fad740711f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:56.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.648+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 >> 172.21.15.86:0/3004290713 conn(0x7fad740f76f0 msgr2=0x7fad741030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:56.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.648+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 shutdown_connections 2024-06-06T08:22:56.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:22:56.648+0000 7fad7adfa640 1 -- 172.21.15.86:0/3004290713 wait complete. 2024-06-06T08:22:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:56 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3004290713' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-06T08:22:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:56 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:56 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:56 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:56 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:56 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:56 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:56 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-06T08:22:57.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:56 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:22:57.218 INFO:tasks.cephadm:Adding mon.smithi086 on smithi086 2024-06-06T08:22:57.218 INFO:tasks.cephadm:Adding mon.smithi154 on smithi154 2024-06-06T08:22:57.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch apply mon '2;smithi086:172.21.15.86=smithi086;smithi154:172.21.15.154=smithi154' 2024-06-06T08:22:57.467 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:22:57.515 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:22:58.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:57 smithi086 ceph-mon[29114]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:22:58.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:57 smithi086 ceph-mon[29114]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-06T08:22:58.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:57 smithi086 ceph-mon[29114]: Deploying daemon grafana.smithi086 on smithi086 2024-06-06T08:22:58.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:57 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3004290713' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-06T08:22:58.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:57 smithi086 ceph-mon[29114]: osdmap e4: 0 total, 0 up, 0 in 2024-06-06T08:22:59.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.068+0000 7f884e681640 1 -- 172.21.15.154:0/3025422510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 msgr2=0x7f88480fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:59.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.068+0000 7f884e681640 1 --2- 172.21.15.154:0/3025422510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 0x7f88480fc320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8838004850 tx=0x7f8838009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:59.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.068+0000 7f884e681640 1 -- 172.21.15.154:0/3025422510 shutdown_connections 2024-06-06T08:22:59.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.068+0000 7f884e681640 1 --2- 172.21.15.154:0/3025422510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 0x7f88480fc320 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:59.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.068+0000 7f884e681640 1 -- 172.21.15.154:0/3025422510 >> 172.21.15.154:0/3025422510 conn(0x7f88480f76f0 msgr2=0x7f88480f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:59.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.068+0000 7f884e681640 1 -- 172.21.15.154:0/3025422510 shutdown_connections 2024-06-06T08:22:59.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.068+0000 7f884e681640 1 -- 172.21.15.154:0/3025422510 wait complete. 2024-06-06T08:22:59.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.069+0000 7f884e681640 1 Processor -- start 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.069+0000 7f884e681640 1 -- start start 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.069+0000 7f884e681640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 0x7f88480711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.069+0000 7f884e681640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8848071730 con 0x7f88480fbf10 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.069+0000 7f8847fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 0x7f88480711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.069+0000 7f8847fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 0x7f88480711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:42310/0 (socket says 172.21.15.154:42310) 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.069+0000 7f8847fff640 1 -- 172.21.15.154:0/381908861 learned_addr learned my addr 172.21.15.154:0/381908861 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.069+0000 7f8847fff640 1 -- 172.21.15.154:0/381908861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8838005040 con 0x7f88480fbf10 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.069+0000 7f8847fff640 1 --2- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 0x7f88480711f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8838004820 tx=0x7f8838009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.070+0000 7f88457fa640 1 -- 172.21.15.154:0/381908861 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8838032650 con 0x7f88480fbf10 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.070+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f884806f7c0 con 0x7f88480fbf10 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.070+0000 7f88457fa640 1 -- 172.21.15.154:0/381908861 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8838032c80 con 0x7f88480fbf10 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.070+0000 7f88457fa640 1 -- 172.21.15.154:0/381908861 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f883803eda0 con 0x7f88480fbf10 2024-06-06T08:22:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.070+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f884806fc90 con 0x7f88480fbf10 2024-06-06T08:22:59.071 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.070+0000 7f88457fa640 1 -- 172.21.15.154:0/381908861 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f8838030020 con 0x7f88480fbf10 2024-06-06T08:22:59.071 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.071+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f884805ead0 con 0x7f88480fbf10 2024-06-06T08:22:59.071 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.071+0000 7f88457fa640 1 --2- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8834034900 0x7f8834036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:22:59.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.071+0000 7f88457fa640 1 -- 172.21.15.154:0/381908861 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8838066a20 con 0x7f88480fbf10 2024-06-06T08:22:59.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.071+0000 7f883ffff640 1 --2- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8834034900 0x7f8834036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:22:59.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.071+0000 7f883ffff640 1 --2- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8834034900 0x7f8834036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8830006fd0 tx=0x7f8830006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:22:59.078 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.079+0000 7f88457fa640 1 -- 172.21.15.154:0/381908861 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8838033420 con 0x7f88480fbf10 2024-06-06T08:22:59.251 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.252+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 --> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi086:172.21.15.86=smithi086;smithi154:172.21.15.154=smithi154", "target": ["mon-mgr", ""]}) v1 -- 0x7f8848105a70 con 0x7f8834034900 2024-06-06T08:22:59.256 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.257+0000 7f88457fa640 1 -- 172.21.15.154:0/381908861 <== mgr.14160 v2:172.21.15.86:6800/2470791934 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8848105a70 con 0x7f8834034900 2024-06-06T08:22:59.256 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled mon update... 2024-06-06T08:22:59.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8834034900 msgr2=0x7f8834036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:59.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 --2- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8834034900 0x7f8834036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8830006fd0 tx=0x7f8830006db0 comp rx=0 tx=0).stop 2024-06-06T08:22:59.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 msgr2=0x7f88480711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:22:59.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 --2- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 0x7f88480711f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8838004820 tx=0x7f8838009da0 comp rx=0 tx=0).stop 2024-06-06T08:22:59.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 shutdown_connections 2024-06-06T08:22:59.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 --2- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f8834034900 0x7f8834036db0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:59.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 --2- 172.21.15.154:0/381908861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88480fbf10 0x7f88480711f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:22:59.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 >> 172.21.15.154:0/381908861 conn(0x7f88480f76f0 msgr2=0x7f88481030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:22:59.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 shutdown_connections 2024-06-06T08:22:59.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:22:59.259+0000 7f884e681640 1 -- 172.21.15.154:0/381908861 wait complete. 2024-06-06T08:22:59.791 DEBUG:teuthology.orchestra.run.smithi154:mon.smithi154> sudo journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi154.service 2024-06-06T08:22:59.794 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:22:59.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:00.071 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:00.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:59 smithi086 ceph-mon[29114]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:00.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:22:59 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:00.119 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:01.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:00 smithi086 ceph-mon[29114]: from='client.14198 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi086:172.21.15.86=smithi086;smithi154:172.21.15.154=smithi154", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:23:01.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:00 smithi086 ceph-mon[29114]: Saving service mon spec with placement smithi086:172.21.15.86=smithi086;smithi154:172.21.15.154=smithi154;count:2 2024-06-06T08:23:01.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:00 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.544+0000 7f79f8866640 1 -- 172.21.15.154:0/1810198369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 msgr2=0x7f79f40fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.544+0000 7f79f8866640 1 --2- 172.21.15.154:0/1810198369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 0x7f79f40fb450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f79e4004850 tx=0x7f79e4009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.544+0000 7f79f8866640 1 -- 172.21.15.154:0/1810198369 shutdown_connections 2024-06-06T08:23:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.544+0000 7f79f8866640 1 --2- 172.21.15.154:0/1810198369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 0x7f79f40fb450 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.544+0000 7f79f8866640 1 -- 172.21.15.154:0/1810198369 >> 172.21.15.154:0/1810198369 conn(0x7f79f40f6820 msgr2=0x7f79f40f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.544+0000 7f79f8866640 1 -- 172.21.15.154:0/1810198369 shutdown_connections 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.544+0000 7f79f8866640 1 -- 172.21.15.154:0/1810198369 wait complete. 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79f8866640 1 Processor -- start 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79f8866640 1 -- start start 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79f8866640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 0x7f79f410c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79f8866640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f410cc70 con 0x7f79f40fb040 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79f259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 0x7f79f410c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79f259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 0x7f79f410c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:42334/0 (socket says 172.21.15.154:42334) 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79f259c640 1 -- 172.21.15.154:0/669020006 learned_addr learned my addr 172.21.15.154:0/669020006 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79f259c640 1 -- 172.21.15.154:0/669020006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e4005040 con 0x7f79f40fb040 2024-06-06T08:23:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79f259c640 1 --2- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 0x7f79f410c730 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f79e4004820 tx=0x7f79e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.545+0000 7f79eb7fe640 1 -- 172.21.15.154:0/669020006 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79e4032650 con 0x7f79f40fb040 2024-06-06T08:23:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.546+0000 7f79eb7fe640 1 -- 172.21.15.154:0/669020006 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79e4032c80 con 0x7f79f40fb040 2024-06-06T08:23:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.546+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f410ce70 con 0x7f79f40fb040 2024-06-06T08:23:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.546+0000 7f79eb7fe640 1 -- 172.21.15.154:0/669020006 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79e403eda0 con 0x7f79f40fb040 2024-06-06T08:23:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.546+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f410da00 con 0x7f79f40fb040 2024-06-06T08:23:01.551 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.546+0000 7f79eb7fe640 1 -- 172.21.15.154:0/669020006 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f79e4030020 con 0x7f79f40fb040 2024-06-06T08:23:01.551 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.547+0000 7f79eb7fe640 1 --2- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f79e0034950 0x7f79e0036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:01.551 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.547+0000 7f79eb7fe640 1 -- 172.21.15.154:0/669020006 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f79e40669c0 con 0x7f79f40fb040 2024-06-06T08:23:01.551 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.547+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f41015b0 con 0x7f79f40fb040 2024-06-06T08:23:01.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.547+0000 7f79e9d9b640 1 --2- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f79e0034950 0x7f79e0036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:01.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.547+0000 7f79e9d9b640 1 --2- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f79e0034950 0x7f79e0036e00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f79dc006fd0 tx=0x7f79dc006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:01.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.552+0000 7f79eb7fe640 1 -- 172.21.15.154:0/669020006 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f79e4033420 con 0x7f79f40fb040 2024-06-06T08:23:01.776 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.777+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79f405ead0 con 0x7f79f40fb040 2024-06-06T08:23:01.776 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.777+0000 7f79eb7fe640 1 -- 172.21.15.154:0/669020006 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f79f405ead0 con 0x7f79f40fb040 2024-06-06T08:23:01.777 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:01.777 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:01.777 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:01.778 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f79e0034950 msgr2=0x7f79e0036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:01.778 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 --2- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f79e0034950 0x7f79e0036e00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f79dc006fd0 tx=0x7f79dc006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:01.779 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 msgr2=0x7f79f410c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:01.779 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 --2- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 0x7f79f410c730 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f79e4004820 tx=0x7f79e4009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:01.779 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 shutdown_connections 2024-06-06T08:23:01.779 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 --2- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f79e0034950 0x7f79e0036e00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:01.779 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 --2- 172.21.15.154:0/669020006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79f40fb040 0x7f79f410c730 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:01.779 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 >> 172.21.15.154:0/669020006 conn(0x7f79f40f6820 msgr2=0x7f79f40646f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:01.779 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 shutdown_connections 2024-06-06T08:23:01.779 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:01.779+0000 7f79f8866640 1 -- 172.21.15.154:0/669020006 wait complete. 2024-06-06T08:23:02.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:01 smithi086 ceph-mon[29114]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:03.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:02 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/669020006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:03.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:03.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:03.561 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:03.610 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:04.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:03 smithi086 ceph-mon[29114]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:05.160 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.156+0000 7fe13697b640 1 -- 172.21.15.154:0/4036170831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 msgr2=0x7fe1300fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:05.160 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.156+0000 7fe13697b640 1 --2- 172.21.15.154:0/4036170831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 0x7fe1300fb2e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe120004850 tx=0x7fe120009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:05.160 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.156+0000 7fe13697b640 1 -- 172.21.15.154:0/4036170831 shutdown_connections 2024-06-06T08:23:05.160 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.156+0000 7fe13697b640 1 --2- 172.21.15.154:0/4036170831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 0x7fe1300fb2e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.156+0000 7fe13697b640 1 -- 172.21.15.154:0/4036170831 >> 172.21.15.154:0/4036170831 conn(0x7fe1300f65f0 msgr2=0x7fe1300f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.156+0000 7fe13697b640 1 -- 172.21.15.154:0/4036170831 shutdown_connections 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.156+0000 7fe13697b640 1 -- 172.21.15.154:0/4036170831 wait complete. 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.157+0000 7fe13697b640 1 Processor -- start 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.157+0000 7fe13697b640 1 -- start start 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.157+0000 7fe13697b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 0x7fe13010b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.157+0000 7fe13697b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13010b970 con 0x7fe1300faed0 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.157+0000 7fe12ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 0x7fe13010b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.157+0000 7fe12ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 0x7fe13010b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:42358/0 (socket says 172.21.15.154:42358) 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.157+0000 7fe12ffff640 1 -- 172.21.15.154:0/1986710228 learned_addr learned my addr 172.21.15.154:0/1986710228 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.158+0000 7fe12ffff640 1 -- 172.21.15.154:0/1986710228 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe120005040 con 0x7fe1300faed0 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.158+0000 7fe12ffff640 1 --2- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 0x7fe13010b430 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe120004820 tx=0x7fe120009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:05.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.158+0000 7fe12cff9640 1 -- 172.21.15.154:0/1986710228 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe120032650 con 0x7fe1300faed0 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.158+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe13010bb70 con 0x7fe1300faed0 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.158+0000 7fe12cff9640 1 -- 172.21.15.154:0/1986710228 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe120032c80 con 0x7fe1300faed0 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.158+0000 7fe12cff9640 1 -- 172.21.15.154:0/1986710228 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe12003eda0 con 0x7fe1300faed0 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.158+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe13010c700 con 0x7fe1300faed0 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.159+0000 7fe12cff9640 1 -- 172.21.15.154:0/1986710228 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fe120030020 con 0x7fe1300faed0 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.159+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1301013a0 con 0x7fe1300faed0 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.159+0000 7fe12cff9640 1 --2- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe118034900 0x7fe118036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.159+0000 7fe12cff9640 1 -- 172.21.15.154:0/1986710228 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe1200669e0 con 0x7fe1300faed0 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.159+0000 7fe12f7fe640 1 --2- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe118034900 0x7fe118036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:05.162 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.160+0000 7fe12f7fe640 1 --2- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe118034900 0x7fe118036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe124006fd0 tx=0x7fe124006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:05.167 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.168+0000 7fe12cff9640 1 -- 172.21.15.154:0/1986710228 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe12003c510 con 0x7fe1300faed0 2024-06-06T08:23:05.393 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.394+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe13010d100 con 0x7fe1300faed0 2024-06-06T08:23:05.393 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.394+0000 7fe12cff9640 1 -- 172.21.15.154:0/1986710228 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe12003c330 con 0x7fe1300faed0 2024-06-06T08:23:05.394 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:05.394 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:05.394 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:05.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe118034900 msgr2=0x7fe118036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:05.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 --2- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe118034900 0x7fe118036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe124006fd0 tx=0x7fe124006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:05.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 msgr2=0x7fe13010b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:05.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 --2- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 0x7fe13010b430 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe120004820 tx=0x7fe120009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:05.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 shutdown_connections 2024-06-06T08:23:05.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 --2- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe118034900 0x7fe118036db0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:05.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 --2- 172.21.15.154:0/1986710228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1300faed0 0x7fe13010b430 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:05.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 >> 172.21.15.154:0/1986710228 conn(0x7fe1300f65f0 msgr2=0x7fe130101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:05.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 shutdown_connections 2024-06-06T08:23:05.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:05.396+0000 7fe13697b640 1 -- 172.21.15.154:0/1986710228 wait complete. 2024-06-06T08:23:06.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:05 smithi086 ceph-mon[29114]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:06.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:05 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/1986710228' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:06.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:06.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:07.226 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:07.905 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:07 smithi086 ceph-mon[29114]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:08.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.585+0000 7fb95dd3c640 1 -- 172.21.15.154:0/2458852591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 msgr2=0x7fb9580fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:08.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.585+0000 7fb95dd3c640 1 --2- 172.21.15.154:0/2458852591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 0x7fb9580fc320 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb948004850 tx=0x7fb948009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:08.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.585+0000 7fb95dd3c640 1 -- 172.21.15.154:0/2458852591 shutdown_connections 2024-06-06T08:23:08.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.585+0000 7fb95dd3c640 1 --2- 172.21.15.154:0/2458852591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 0x7fb9580fc320 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:08.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.585+0000 7fb95dd3c640 1 -- 172.21.15.154:0/2458852591 >> 172.21.15.154:0/2458852591 conn(0x7fb9580f76f0 msgr2=0x7fb9580f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.585+0000 7fb95dd3c640 1 -- 172.21.15.154:0/2458852591 shutdown_connections 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.585+0000 7fb95dd3c640 1 -- 172.21.15.154:0/2458852591 wait complete. 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.585+0000 7fb95dd3c640 1 Processor -- start 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.585+0000 7fb95dd3c640 1 -- start start 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.586+0000 7fb95dd3c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 0x7fb9580711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.586+0000 7fb95dd3c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb958071730 con 0x7fb9580fbf10 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.586+0000 7fb9577fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 0x7fb9580711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.586+0000 7fb9577fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 0x7fb9580711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:34498/0 (socket says 172.21.15.154:34498) 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.586+0000 7fb9577fe640 1 -- 172.21.15.154:0/3595258608 learned_addr learned my addr 172.21.15.154:0/3595258608 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:08.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.586+0000 7fb9577fe640 1 -- 172.21.15.154:0/3595258608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb948005040 con 0x7fb9580fbf10 2024-06-06T08:23:08.587 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.586+0000 7fb9577fe640 1 --2- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 0x7fb9580711f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb948004820 tx=0x7fb948009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.586+0000 7fb93ffff640 1 -- 172.21.15.154:0/3595258608 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb948032650 con 0x7fb9580fbf10 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.587+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95806f7c0 con 0x7fb9580fbf10 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.587+0000 7fb93ffff640 1 -- 172.21.15.154:0/3595258608 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb948032c80 con 0x7fb9580fbf10 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.587+0000 7fb93ffff640 1 -- 172.21.15.154:0/3595258608 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb94803eda0 con 0x7fb9580fbf10 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.587+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95806fc90 con 0x7fb9580fbf10 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.587+0000 7fb93ffff640 1 -- 172.21.15.154:0/3595258608 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fb948030020 con 0x7fb9580fbf10 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.587+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95805ead0 con 0x7fb9580fbf10 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.588+0000 7fb93ffff640 1 --2- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fb940034900 0x7fb940036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.588+0000 7fb93ffff640 1 -- 172.21.15.154:0/3595258608 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb9480669a0 con 0x7fb9580fbf10 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.588+0000 7fb956ffd640 1 --2- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fb940034900 0x7fb940036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.589+0000 7fb956ffd640 1 --2- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fb940034900 0x7fb940036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb95000ad90 tx=0x7fb950009410 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:08.593 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.594+0000 7fb93ffff640 1 -- 172.21.15.154:0/3595258608 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb9480327b0 con 0x7fb9580fbf10 2024-06-06T08:23:08.820 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.821+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb95804b480 con 0x7fb9580fbf10 2024-06-06T08:23:08.821 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.821+0000 7fb93ffff640 1 -- 172.21.15.154:0/3595258608 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb94803c330 con 0x7fb9580fbf10 2024-06-06T08:23:08.821 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:08.821 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:08.821 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:08.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fb940034900 msgr2=0x7fb940036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:08.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 --2- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fb940034900 0x7fb940036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb95000ad90 tx=0x7fb950009410 comp rx=0 tx=0).stop 2024-06-06T08:23:08.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 msgr2=0x7fb9580711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:08.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 --2- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 0x7fb9580711f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb948004820 tx=0x7fb948009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:08.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 shutdown_connections 2024-06-06T08:23:08.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 --2- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fb940034900 0x7fb940036db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:08.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 --2- 172.21.15.154:0/3595258608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9580fbf10 0x7fb9580711f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:08.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 >> 172.21.15.154:0/3595258608 conn(0x7fb9580f76f0 msgr2=0x7fb9581030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:08.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 shutdown_connections 2024-06-06T08:23:08.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:08.824+0000 7fb95dd3c640 1 -- 172.21.15.154:0/3595258608 wait complete. 2024-06-06T08:23:09.789 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:09 smithi086 ceph-mon[29114]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:09.789 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:09 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/3595258608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:09.789 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:09 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:09.789 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:09 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:09.789 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:09 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:09.789 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:09 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:09.789 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:09 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:09.789 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:09 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:10.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:10.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:10.718 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:10.765 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:10.976 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:10 smithi086 ceph-mon[29114]: Deploying daemon prometheus.smithi086 on smithi086 2024-06-06T08:23:10.977 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:10 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:12.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:11 smithi086 ceph-mon[29114]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:12.086 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.086+0000 7fe7c93f7640 1 -- 172.21.15.154:0/1207147927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 msgr2=0x7fe7c40fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.086+0000 7fe7c93f7640 1 --2- 172.21.15.154:0/1207147927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 0x7fe7c40fb280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4004850 tx=0x7fe7b4009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c93f7640 1 -- 172.21.15.154:0/1207147927 shutdown_connections 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c93f7640 1 --2- 172.21.15.154:0/1207147927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 0x7fe7c40fb280 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c93f7640 1 -- 172.21.15.154:0/1207147927 >> 172.21.15.154:0/1207147927 conn(0x7fe7c40f65e0 msgr2=0x7fe7c40f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c93f7640 1 -- 172.21.15.154:0/1207147927 shutdown_connections 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c93f7640 1 -- 172.21.15.154:0/1207147927 wait complete. 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c93f7640 1 Processor -- start 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c93f7640 1 -- start start 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c93f7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 0x7fe7c410c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c93f7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c410cab0 con 0x7fe7c40fae70 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 0x7fe7c410c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 0x7fe7c410c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:34518/0 (socket says 172.21.15.154:34518) 2024-06-06T08:23:12.088 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.087+0000 7fe7c2ffd640 1 -- 172.21.15.154:0/4260224485 learned_addr learned my addr 172.21.15.154:0/4260224485 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:12.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.088+0000 7fe7c2ffd640 1 -- 172.21.15.154:0/4260224485 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b4005040 con 0x7fe7c40fae70 2024-06-06T08:23:12.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.088+0000 7fe7c2ffd640 1 --2- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 0x7fe7c410c570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4004820 tx=0x7fe7b4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:12.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.088+0000 7fe7bb7fe640 1 -- 172.21.15.154:0/4260224485 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b4032650 con 0x7fe7c40fae70 2024-06-06T08:23:12.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.088+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c410ccb0 con 0x7fe7c40fae70 2024-06-06T08:23:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.088+0000 7fe7bb7fe640 1 -- 172.21.15.154:0/4260224485 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7b4032c80 con 0x7fe7c40fae70 2024-06-06T08:23:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.088+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c410d840 con 0x7fe7c40fae70 2024-06-06T08:23:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.088+0000 7fe7bb7fe640 1 -- 172.21.15.154:0/4260224485 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b4033d60 con 0x7fe7c40fae70 2024-06-06T08:23:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.089+0000 7fe7bb7fe640 1 -- 172.21.15.154:0/4260224485 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fe7b4030020 con 0x7fe7c40fae70 2024-06-06T08:23:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.089+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c4101390 con 0x7fe7c40fae70 2024-06-06T08:23:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.089+0000 7fe7bb7fe640 1 --2- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe7b0034900 0x7fe7b0036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.089+0000 7fe7bb7fe640 1 -- 172.21.15.154:0/4260224485 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe7b4066770 con 0x7fe7c40fae70 2024-06-06T08:23:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.090+0000 7fe7bbfff640 1 --2- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe7b0034900 0x7fe7b0036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:12.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.090+0000 7fe7bbfff640 1 --2- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe7b0034900 0x7fe7b0036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac006fd0 tx=0x7fe7ac006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:12.097 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.098+0000 7fe7bb7fe640 1 -- 172.21.15.154:0/4260224485 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe7b4032df0 con 0x7fe7c40fae70 2024-06-06T08:23:12.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.322+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7c410da80 con 0x7fe7c40fae70 2024-06-06T08:23:12.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.323+0000 7fe7bb7fe640 1 -- 172.21.15.154:0/4260224485 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe7b403c380 con 0x7fe7c40fae70 2024-06-06T08:23:12.322 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:12.322 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:12.322 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:12.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe7b0034900 msgr2=0x7fe7b0036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:12.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 --2- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe7b0034900 0x7fe7b0036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac006fd0 tx=0x7fe7ac006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:12.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 msgr2=0x7fe7c410c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:12.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 --2- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 0x7fe7c410c570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4004820 tx=0x7fe7b4009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:12.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 shutdown_connections 2024-06-06T08:23:12.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 --2- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7fe7b0034900 0x7fe7b0036db0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:12.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 --2- 172.21.15.154:0/4260224485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7c40fae70 0x7fe7c410c570 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:12.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 >> 172.21.15.154:0/4260224485 conn(0x7fe7c40f65e0 msgr2=0x7fe7c4065260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:12.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 shutdown_connections 2024-06-06T08:23:12.325 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:12.325+0000 7fe7c93f7640 1 -- 172.21.15.154:0/4260224485 wait complete. 2024-06-06T08:23:13.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:12 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/4260224485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:13.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:13.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:14.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:13 smithi086 ceph-mon[29114]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:14.102 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:14.150 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:15.494 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.489+0000 7f1aacc3f640 1 -- 172.21.15.154:0/2950701698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 msgr2=0x7f1aa80fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:15.494 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.489+0000 7f1aacc3f640 1 --2- 172.21.15.154:0/2950701698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 0x7f1aa80fb450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009bd0 tx=0x7f1a98004d10 comp rx=0 tx=0).stop 2024-06-06T08:23:15.494 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.489+0000 7f1aacc3f640 1 -- 172.21.15.154:0/2950701698 shutdown_connections 2024-06-06T08:23:15.494 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.489+0000 7f1aacc3f640 1 --2- 172.21.15.154:0/2950701698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 0x7f1aa80fb450 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.489+0000 7f1aacc3f640 1 -- 172.21.15.154:0/2950701698 >> 172.21.15.154:0/2950701698 conn(0x7f1aa80f6820 msgr2=0x7f1aa80f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.489+0000 7f1aacc3f640 1 -- 172.21.15.154:0/2950701698 shutdown_connections 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.489+0000 7f1aacc3f640 1 -- 172.21.15.154:0/2950701698 wait complete. 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.489+0000 7f1aacc3f640 1 Processor -- start 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.490+0000 7f1aacc3f640 1 -- start start 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.490+0000 7f1aacc3f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 0x7f1aa810c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.490+0000 7f1aacc3f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa810cc70 con 0x7f1aa80fb040 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.490+0000 7f1aa659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 0x7f1aa810c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.490+0000 7f1aa659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 0x7f1aa810c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:34542/0 (socket says 172.21.15.154:34542) 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.490+0000 7f1aa659c640 1 -- 172.21.15.154:0/277103726 learned_addr learned my addr 172.21.15.154:0/277103726 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.490+0000 7f1aa659c640 1 -- 172.21.15.154:0/277103726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a980098b0 con 0x7f1aa80fb040 2024-06-06T08:23:15.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.491+0000 7f1aa659c640 1 --2- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 0x7f1aa810c730 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1a98003400 tx=0x7f1a98004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.491+0000 7f1a96ffd640 1 -- 172.21.15.154:0/277103726 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a980329b0 con 0x7f1aa80fb040 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.491+0000 7f1a96ffd640 1 -- 172.21.15.154:0/277103726 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a9803d420 con 0x7f1aa80fb040 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.491+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa810ce70 con 0x7f1aa80fb040 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.491+0000 7f1a96ffd640 1 -- 172.21.15.154:0/277103726 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a98031590 con 0x7f1aa80fb040 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.491+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa810d2c0 con 0x7f1aa80fb040 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.492+0000 7f1a96ffd640 1 -- 172.21.15.154:0/277103726 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f1a9802f030 con 0x7f1aa80fb040 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.492+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa805ead0 con 0x7f1aa80fb040 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.492+0000 7f1a96ffd640 1 --2- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f1a900348b0 0x7f1a90036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.492+0000 7f1a96ffd640 1 -- 172.21.15.154:0/277103726 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1a98067be0 con 0x7f1aa80fb040 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.492+0000 7f1aa5d9b640 1 --2- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f1a900348b0 0x7f1a90036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:15.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.493+0000 7f1aa5d9b640 1 --2- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f1a900348b0 0x7f1a90036d60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c006fd0 tx=0x7f1a9c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:15.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.497+0000 7f1a96ffd640 1 -- 172.21.15.154:0/277103726 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1a9804e7c0 con 0x7f1aa80fb040 2024-06-06T08:23:15.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.723+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1aa8100f70 con 0x7f1aa80fb040 2024-06-06T08:23:15.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.724+0000 7f1a96ffd640 1 -- 172.21.15.154:0/277103726 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1aa8100f70 con 0x7f1aa80fb040 2024-06-06T08:23:15.723 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:15.723 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:15.724 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:15.725 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f1a900348b0 msgr2=0x7f1a90036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:15.725 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 --2- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f1a900348b0 0x7f1a90036d60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c006fd0 tx=0x7f1a9c006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:15.725 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 msgr2=0x7f1aa810c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:15.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 --2- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 0x7f1aa810c730 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1a98003400 tx=0x7f1a98004d10 comp rx=0 tx=0).stop 2024-06-06T08:23:15.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 shutdown_connections 2024-06-06T08:23:15.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 --2- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f1a900348b0 0x7f1a90036d60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:15.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 --2- 172.21.15.154:0/277103726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1aa80fb040 0x7f1aa810c730 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:15.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 >> 172.21.15.154:0/277103726 conn(0x7f1aa80f6820 msgr2=0x7f1aa80646f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:15.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 shutdown_connections 2024-06-06T08:23:15.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:15.726+0000 7f1aacc3f640 1 -- 172.21.15.154:0/277103726 wait complete. 2024-06-06T08:23:16.066 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:15 smithi086 ceph-mon[29114]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:16.949 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:16 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/277103726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:17.303 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:17.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:17.554 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:17.602 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:17.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:17 smithi086 ceph-mon[29114]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:18.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.902+0000 7f2609b16640 1 -- 172.21.15.154:0/3102169902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 msgr2=0x7f26040fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:18.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.902+0000 7f2609b16640 1 --2- 172.21.15.154:0/3102169902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 0x7f26040fb280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009bd0 tx=0x7f25f4004d10 comp rx=0 tx=0).stop 2024-06-06T08:23:18.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.902+0000 7f2609b16640 1 -- 172.21.15.154:0/3102169902 shutdown_connections 2024-06-06T08:23:18.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.902+0000 7f2609b16640 1 --2- 172.21.15.154:0/3102169902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 0x7f26040fb280 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:18.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.902+0000 7f2609b16640 1 -- 172.21.15.154:0/3102169902 >> 172.21.15.154:0/3102169902 conn(0x7f26040f65e0 msgr2=0x7f26040f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:18.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.902+0000 7f2609b16640 1 -- 172.21.15.154:0/3102169902 shutdown_connections 2024-06-06T08:23:18.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.902+0000 7f2609b16640 1 -- 172.21.15.154:0/3102169902 wait complete. 2024-06-06T08:23:18.903 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.902+0000 7f2609b16640 1 Processor -- start 2024-06-06T08:23:18.903 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f2609b16640 1 -- start start 2024-06-06T08:23:18.903 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f2609b16640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 0x7f260410b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:18.903 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f2609b16640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260410ba50 con 0x7f26040fae70 2024-06-06T08:23:18.903 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f26037fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 0x7f260410b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:18.903 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f26037fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 0x7f260410b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:57788/0 (socket says 172.21.15.154:57788) 2024-06-06T08:23:18.903 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f26037fe640 1 -- 172.21.15.154:0/2753124217 learned_addr learned my addr 172.21.15.154:0/2753124217 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:18.904 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f26037fe640 1 -- 172.21.15.154:0/2753124217 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f40098b0 con 0x7f26040fae70 2024-06-06T08:23:18.904 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f26037fe640 1 --2- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 0x7f260410b510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009ba0 tx=0x7f25f4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f25ebfff640 1 -- 172.21.15.154:0/2753124217 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f40329b0 con 0x7f26040fae70 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.903+0000 7f25ebfff640 1 -- 172.21.15.154:0/2753124217 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25f403d420 con 0x7f26040fae70 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.904+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260410bc50 con 0x7f26040fae70 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.904+0000 7f25ebfff640 1 -- 172.21.15.154:0/2753124217 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f4031590 con 0x7f26040fae70 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.904+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260410c0a0 con 0x7f26040fae70 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.904+0000 7f25ebfff640 1 -- 172.21.15.154:0/2753124217 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f25f402f030 con 0x7f26040fae70 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.904+0000 7f25ebfff640 1 --2- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f25ec0348b0 0x7f25ec036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.905+0000 7f25ebfff640 1 -- 172.21.15.154:0/2753124217 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f25f4067cb0 con 0x7f26040fae70 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.905+0000 7f2602ffd640 1 --2- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f25ec0348b0 0x7f25ec036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.905+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260405ead0 con 0x7f26040fae70 2024-06-06T08:23:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.905+0000 7f2602ffd640 1 --2- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f25ec0348b0 0x7f25ec036d60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25f8006fd0 tx=0x7f25f8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:18.908 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:18.909+0000 7f25ebfff640 1 -- 172.21.15.154:0/2753124217 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f25f4041070 con 0x7f26040fae70 2024-06-06T08:23:19.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.137+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f260404b480 con 0x7f26040fae70 2024-06-06T08:23:19.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.137+0000 7f25ebfff640 1 -- 172.21.15.154:0/2753124217 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f25f404e710 con 0x7f26040fae70 2024-06-06T08:23:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:19.137 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:19.137 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f25ec0348b0 msgr2=0x7f25ec036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 --2- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f25ec0348b0 0x7f25ec036d60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25f8006fd0 tx=0x7f25f8006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 msgr2=0x7f260410b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 --2- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 0x7f260410b510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009ba0 tx=0x7f25f4004d10 comp rx=0 tx=0).stop 2024-06-06T08:23:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 shutdown_connections 2024-06-06T08:23:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 --2- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f25ec0348b0 0x7f25ec036d60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 --2- 172.21.15.154:0/2753124217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26040fae70 0x7f260410b510 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 >> 172.21.15.154:0/2753124217 conn(0x7f26040f65e0 msgr2=0x7f2604101fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 shutdown_connections 2024-06-06T08:23:19.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:19.140+0000 7f2609b16640 1 -- 172.21.15.154:0/2753124217 wait complete. 2024-06-06T08:23:20.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:19 smithi086 ceph-mon[29114]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:20.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:19 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/2753124217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:20.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:19 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:20.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:19 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:20.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:19 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:20.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:19 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-06T08:23:20.729 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:20.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:21.069 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:21.116 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:21.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:21 smithi086 ceph-mon[29114]: from='mgr.14160 172.21.15.86:0/1433421676' entity='mgr.smithi086.mfkgze' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-06T08:23:21.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:21 smithi086 ceph-mon[29114]: mgrmap e14: smithi086.mfkgze(active, since 54s) 2024-06-06T08:23:22.602 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.602+0000 7f658e9e3640 1 -- 172.21.15.154:0/849088746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 msgr2=0x7f65880fc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.602+0000 7f658e9e3640 1 --2- 172.21.15.154:0/849088746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 0x7f65880fc840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6570004850 tx=0x7f6570009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.602+0000 7f658e9e3640 1 -- 172.21.15.154:0/849088746 shutdown_connections 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.602+0000 7f658e9e3640 1 --2- 172.21.15.154:0/849088746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 0x7f65880fc840 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.602+0000 7f658e9e3640 1 -- 172.21.15.154:0/849088746 >> 172.21.15.154:0/849088746 conn(0x7f65880f5d70 msgr2=0x7f65880f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.602+0000 7f658e9e3640 1 -- 172.21.15.154:0/849088746 shutdown_connections 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.602+0000 7f658e9e3640 1 -- 172.21.15.154:0/849088746 wait complete. 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.603+0000 7f658e9e3640 1 Processor -- start 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.603+0000 7f658e9e3640 1 -- start start 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.603+0000 7f658e9e3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 0x7f65880fabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.603+0000 7f658e9e3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658810d090 con 0x7f65880fc430 2024-06-06T08:23:22.604 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.603+0000 7f6587fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 0x7f65880fabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:22.604 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.603+0000 7f6587fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 0x7f65880fabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:57810/0 (socket says 172.21.15.154:57810) 2024-06-06T08:23:22.604 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.603+0000 7f6587fff640 1 -- 172.21.15.154:0/1440183155 learned_addr learned my addr 172.21.15.154:0/1440183155 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:22.604 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.603+0000 7f6587fff640 1 -- 172.21.15.154:0/1440183155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6570005040 con 0x7f65880fc430 2024-06-06T08:23:22.605 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.604+0000 7f6587fff640 1 --2- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 0x7f65880fabe0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6570004820 tx=0x7f6570009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:22.605 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.604+0000 7f6584ff9640 1 -- 172.21.15.154:0/1440183155 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6570032650 con 0x7f65880fc430 2024-06-06T08:23:22.605 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.604+0000 7f6584ff9640 1 -- 172.21.15.154:0/1440183155 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6570032c80 con 0x7f65880fc430 2024-06-06T08:23:22.605 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.604+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65881116a0 con 0x7f65880fc430 2024-06-06T08:23:22.605 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.604+0000 7f6584ff9640 1 -- 172.21.15.154:0/1440183155 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f657003ed70 con 0x7f65880fc430 2024-06-06T08:23:22.605 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.604+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6588111a50 con 0x7f65880fc430 2024-06-06T08:23:22.606 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.605+0000 7f6584ff9640 1 -- 172.21.15.154:0/1440183155 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 14) v1 ==== 42016+0+0 (secure 0 0 0) 0x7f6570030020 con 0x7f65880fc430 2024-06-06T08:23:22.606 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.605+0000 7f6584ff9640 1 --2- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6574034950 0x7f6574036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:22.606 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.605+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658805ead0 con 0x7f65880fc430 2024-06-06T08:23:22.606 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.605+0000 7f6584ff9640 1 -- 172.21.15.154:0/1440183155 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6570066a50 con 0x7f65880fc430 2024-06-06T08:23:22.606 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.605+0000 7f65877fe640 1 -- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6574034950 msgr2=0x7f6574036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/2470791934 2024-06-06T08:23:22.606 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.605+0000 7f65877fe640 1 --2- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6574034950 0x7f6574036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T08:23:22.609 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.610+0000 7f6584ff9640 1 -- 172.21.15.154:0/1440183155 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f657003c920 con 0x7f65880fc430 2024-06-06T08:23:22.805 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.806+0000 7f65877fe640 1 -- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6574034950 msgr2=0x7f6574036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/2470791934 2024-06-06T08:23:22.805 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.806+0000 7f65877fe640 1 --2- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6574034950 0x7f6574036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T08:23:22.839 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.840+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f658802a8e0 con 0x7f65880fc430 2024-06-06T08:23:22.840 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.841+0000 7f6584ff9640 1 -- 172.21.15.154:0/1440183155 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6570032e60 con 0x7f65880fc430 2024-06-06T08:23:22.840 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:22.840 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:22.841 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:22.842 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6574034950 msgr2=0x7f6574036e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T08:23:22.842 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 --2- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6574034950 0x7f6574036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:22.842 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 msgr2=0x7f65880fabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:22.842 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 --2- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 0x7f65880fabe0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6570004820 tx=0x7f6570009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:22.842 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 shutdown_connections 2024-06-06T08:23:22.842 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 --2- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:6800/2470791934,v1:172.21.15.86:6801/2470791934] conn(0x7f6574034950 0x7f6574036e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:22.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 --2- 172.21.15.154:0/1440183155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65880fc430 0x7f65880fabe0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:22.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 >> 172.21.15.154:0/1440183155 conn(0x7f65880f5d70 msgr2=0x7f6588101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:22.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 shutdown_connections 2024-06-06T08:23:22.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:22.843+0000 7f658e9e3640 1 -- 172.21.15.154:0/1440183155 wait complete. 2024-06-06T08:23:23.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:22 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/1440183155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:24.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:24.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:24.698 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:24.745 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: Active manager daemon smithi086.mfkgze restarted 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: Activating manager daemon smithi086.mfkgze 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: osdmap e5: 0 total, 0 up, 0 in 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: mgrmap e15: smithi086.mfkgze(active, starting, since 0.00290781s) 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr metadata", "who": "smithi086.mfkgze", "id": "smithi086.mfkgze"}]: dispatch 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: Manager daemon smithi086.mfkgze is now available 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:23:24.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:24.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:24.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:23:26.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:25 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.mfkgze/mirror_snapshot_schedule"}]: dispatch 2024-06-06T08:23:26.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:25 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.mfkgze/trash_purge_schedule"}]: dispatch 2024-06-06T08:23:26.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:25 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:26.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:25 smithi086 ceph-mon[29114]: mgrmap e16: smithi086.mfkgze(active, since 1.00752s) 2024-06-06T08:23:26.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.099+0000 7f8b8efb2640 1 -- 172.21.15.154:0/2663220070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 msgr2=0x7f8b880fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:26.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.099+0000 7f8b8efb2640 1 --2- 172.21.15.154:0/2663220070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 0x7f8b880fb450 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8b78004850 tx=0x7f8b78009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:26.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.099+0000 7f8b8efb2640 1 -- 172.21.15.154:0/2663220070 shutdown_connections 2024-06-06T08:23:26.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.099+0000 7f8b8efb2640 1 --2- 172.21.15.154:0/2663220070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 0x7f8b880fb450 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:26.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.099+0000 7f8b8efb2640 1 -- 172.21.15.154:0/2663220070 >> 172.21.15.154:0/2663220070 conn(0x7f8b880f6820 msgr2=0x7f8b880f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:26.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.099+0000 7f8b8efb2640 1 -- 172.21.15.154:0/2663220070 shutdown_connections 2024-06-06T08:23:26.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.099+0000 7f8b8efb2640 1 -- 172.21.15.154:0/2663220070 wait complete. 2024-06-06T08:23:26.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.099+0000 7f8b8efb2640 1 Processor -- start 2024-06-06T08:23:26.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.100+0000 7f8b8efb2640 1 -- start start 2024-06-06T08:23:26.106 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.100+0000 7f8b8efb2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 0x7f8b88065250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:26.106 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.100+0000 7f8b8efb2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8810f460 con 0x7f8b880fb040 2024-06-06T08:23:26.106 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.100+0000 7f8b8cd4e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 0x7f8b88065250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:26.106 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.100+0000 7f8b8cd4e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 0x7f8b88065250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:57824/0 (socket says 172.21.15.154:57824) 2024-06-06T08:23:26.106 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.100+0000 7f8b8cd4e640 1 -- 172.21.15.154:0/3200082282 learned_addr learned my addr 172.21.15.154:0/3200082282 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:26.106 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.100+0000 7f8b8cd4e640 1 -- 172.21.15.154:0/3200082282 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b78005040 con 0x7f8b880fb040 2024-06-06T08:23:26.106 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.101+0000 7f8b8cd4e640 1 --2- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 0x7f8b88065250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8b78004820 tx=0x7f8b78009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:26.106 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.101+0000 7f8b857fa640 1 -- 172.21.15.154:0/3200082282 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b78032650 con 0x7f8b880fb040 2024-06-06T08:23:26.106 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.101+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8810f5a0 con 0x7f8b880fb040 2024-06-06T08:23:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.101+0000 7f8b857fa640 1 -- 172.21.15.154:0/3200082282 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b78032c80 con 0x7f8b880fb040 2024-06-06T08:23:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.101+0000 7f8b857fa640 1 -- 172.21.15.154:0/3200082282 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b7803eda0 con 0x7f8b880fb040 2024-06-06T08:23:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.101+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8810fa70 con 0x7f8b880fb040 2024-06-06T08:23:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.102+0000 7f8b857fa640 1 -- 172.21.15.154:0/3200082282 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 41867+0+0 (secure 0 0 0) 0x7f8b78030020 con 0x7f8b880fb040 2024-06-06T08:23:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.102+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b881015b0 con 0x7f8b880fb040 2024-06-06T08:23:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.102+0000 7f8b857fa640 1 --2- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b70034790 0x7f8b70036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.102+0000 7f8b857fa640 1 -- 172.21.15.154:0/3200082282 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f8b78066830 con 0x7f8b880fb040 2024-06-06T08:23:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.102+0000 7f8b87fff640 1 --2- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b70034790 0x7f8b70036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.102+0000 7f8b87fff640 1 --2- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b70034790 0x7f8b70036c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c006fd0 tx=0x7f8b7c006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:26.108 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.106+0000 7f8b857fa640 1 -- 172.21.15.154:0/3200082282 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8b78033420 con 0x7f8b880fb040 2024-06-06T08:23:26.339 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.340+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b8805ead0 con 0x7f8b880fb040 2024-06-06T08:23:26.340 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.340+0000 7f8b857fa640 1 -- 172.21.15.154:0/3200082282 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8b7803c330 con 0x7f8b880fb040 2024-06-06T08:23:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:26.340 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:26.341 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:26.346 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b70034790 msgr2=0x7f8b70036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:26.346 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 --2- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b70034790 0x7f8b70036c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c006fd0 tx=0x7f8b7c006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:26.346 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 msgr2=0x7f8b88065250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:26.346 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 --2- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 0x7f8b88065250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8b78004820 tx=0x7f8b78009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:26.346 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 shutdown_connections 2024-06-06T08:23:26.346 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 --2- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b70034790 0x7f8b70036c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:26.346 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 --2- 172.21.15.154:0/3200082282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b880fb040 0x7f8b88065250 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:26.346 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 >> 172.21.15.154:0/3200082282 conn(0x7f8b880f6820 msgr2=0x7f8b881021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:26.347 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 shutdown_connections 2024-06-06T08:23:26.347 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:26.343+0000 7f8b8efb2640 1 -- 172.21.15.154:0/3200082282 wait complete. 2024-06-06T08:23:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:26 smithi086 ceph-mon[29114]: [06/Jun/2024:08:23:25] ENGINE Bus STARTING 2024-06-06T08:23:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:26 smithi086 ceph-mon[29114]: [06/Jun/2024:08:23:25] ENGINE Serving on https://172.21.15.86:7150 2024-06-06T08:23:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:26 smithi086 ceph-mon[29114]: [06/Jun/2024:08:23:25] ENGINE Bus STARTED 2024-06-06T08:23:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:26 smithi086 ceph-mon[29114]: [06/Jun/2024:08:23:25] ENGINE Error in HTTPServer.serve 2024-06-06T08:23:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Traceback (most recent call last): 2024-06-06T08:23:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-06T08:23:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-06T08:23:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-06T08:23:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self._run(expiration_interval) 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: s = self.context.wrap_socket( 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: return self.sslsocket_class._create( 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self.do_handshake() 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-06T08:23:27.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: self._sslobj.do_handshake() 2024-06-06T08:23:27.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-06T08:23:27.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:26 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:27.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:26 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:27.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:26 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/3200082282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:27.868 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:27.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:28.189 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:28.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:28 smithi086 ceph-mon[29114]: mgrmap e17: smithi086.mfkgze(active, since 2s) 2024-06-06T08:23:28.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:28 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:29.076 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T08:23:30.542 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.541+0000 7fe43805b640 1 -- 172.21.15.154:0/207670914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 msgr2=0x7fe4300fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.541+0000 7fe43805b640 1 --2- 172.21.15.154:0/207670914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 0x7fe4300fc320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009bd0 tx=0x7fe42c004d10 comp rx=0 tx=0).stop 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.541+0000 7fe43805b640 1 -- 172.21.15.154:0/207670914 shutdown_connections 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.541+0000 7fe43805b640 1 --2- 172.21.15.154:0/207670914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 0x7fe4300fc320 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.541+0000 7fe43805b640 1 -- 172.21.15.154:0/207670914 >> 172.21.15.154:0/207670914 conn(0x7fe4300f76f0 msgr2=0x7fe4300f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.541+0000 7fe43805b640 1 -- 172.21.15.154:0/207670914 shutdown_connections 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.541+0000 7fe43805b640 1 -- 172.21.15.154:0/207670914 wait complete. 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.542+0000 7fe43805b640 1 Processor -- start 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.542+0000 7fe43805b640 1 -- start start 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.542+0000 7fe43805b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 0x7fe43010c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:30.543 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.542+0000 7fe43805b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43010ca50 con 0x7fe4300fbf10 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe435df7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 0x7fe43010c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe435df7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 0x7fe43010c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:51118/0 (socket says 172.21.15.154:51118) 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe435df7640 1 -- 172.21.15.154:0/2325851800 learned_addr learned my addr 172.21.15.154:0/2325851800 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe435df7640 1 -- 172.21.15.154:0/2325851800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c0098b0 con 0x7fe4300fbf10 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe435df7640 1 --2- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 0x7fe43010c510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009ba0 tx=0x7fe42c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe4267fc640 1 -- 172.21.15.154:0/2325851800 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe42c0329b0 con 0x7fe4300fbf10 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43010cc50 con 0x7fe4300fbf10 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe4267fc640 1 -- 172.21.15.154:0/2325851800 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe42c03d420 con 0x7fe4300fbf10 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe4267fc640 1 -- 172.21.15.154:0/2325851800 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe42c031590 con 0x7fe4300fbf10 2024-06-06T08:23:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.543+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43010d0a0 con 0x7fe4300fbf10 2024-06-06T08:23:30.545 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.544+0000 7fe4267fc640 1 -- 172.21.15.154:0/2325851800 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7fe42c02f030 con 0x7fe4300fbf10 2024-06-06T08:23:30.545 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.544+0000 7fe4267fc640 1 --2- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe41c0349a0 0x7fe41c036e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:30.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.544+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43005ead0 con 0x7fe4300fbf10 2024-06-06T08:23:30.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.544+0000 7fe4267fc640 1 -- 172.21.15.154:0/2325851800 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fe42c067f50 con 0x7fe4300fbf10 2024-06-06T08:23:30.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.545+0000 7fe4355f6640 1 --2- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe41c0349a0 0x7fe41c036e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:30.546 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.545+0000 7fe4355f6640 1 --2- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe41c0349a0 0x7fe41c036e50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe420006fd0 tx=0x7fe420006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:30.548 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.549+0000 7fe4267fc640 1 -- 172.21.15.154:0/2325851800 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe42c04e940 con 0x7fe4300fbf10 2024-06-06T08:23:30.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:30.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:30.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:30.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:30.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:23:30.777 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.778+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe430101cb0 con 0x7fe4300fbf10 2024-06-06T08:23:30.778 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.778+0000 7fe4267fc640 1 -- 172.21.15.154:0/2325851800 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe42c0361c0 con 0x7fe4300fbf10 2024-06-06T08:23:30.778 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:30.778 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:30.779 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.780+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe41c0349a0 msgr2=0x7fe41c036e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.780+0000 7fe43805b640 1 --2- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe41c0349a0 0x7fe41c036e50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe420006fd0 tx=0x7fe420006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.780+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 msgr2=0x7fe43010c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.780+0000 7fe43805b640 1 --2- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 0x7fe43010c510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009ba0 tx=0x7fe42c004d10 comp rx=0 tx=0).stop 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.780+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 shutdown_connections 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.780+0000 7fe43805b640 1 --2- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe41c0349a0 0x7fe41c036e50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.780+0000 7fe43805b640 1 --2- 172.21.15.154:0/2325851800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4300fbf10 0x7fe43010c510 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.781+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 >> 172.21.15.154:0/2325851800 conn(0x7fe4300f76f0 msgr2=0x7fe4301030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.781+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 shutdown_connections 2024-06-06T08:23:30.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:30.781+0000 7fe43805b640 1 -- 172.21.15.154:0/2325851800 wait complete. 2024-06-06T08:23:31.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:31 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/2325851800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:31.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:31.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:31.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:31.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:31.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:23:31.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:31.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: Updating smithi154:/etc/ceph/ceph.conf 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: Updating smithi154:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: Updating smithi086:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:32.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi154", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T08:23:32.586 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi154", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-06T08:23:32.586 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:32.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:32.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:33.158 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T08:23:33.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:33 smithi086 ceph-mon[29114]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-06-06T08:23:33.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:33 smithi086 ceph-mon[29114]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-06T08:23:33.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:33 smithi086 ceph-mon[29114]: Updating smithi154:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-06T08:23:33.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:33 smithi086 ceph-mon[29114]: Updating smithi086:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-06T08:23:33.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:33 smithi086 ceph-mon[29114]: Deploying daemon crash.smithi154 on smithi154 2024-06-06T08:23:34.883 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.871+0000 7f00f0852640 1 -- 172.21.15.154:0/2145656990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 msgr2=0x7f00ec102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:34.883 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.871+0000 7f00f0852640 1 --2- 172.21.15.154:0/2145656990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 0x7f00ec102310 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009bd0 tx=0x7f00e4004d10 comp rx=0 tx=0).stop 2024-06-06T08:23:34.883 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.871+0000 7f00f0852640 1 -- 172.21.15.154:0/2145656990 shutdown_connections 2024-06-06T08:23:34.883 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.871+0000 7f00f0852640 1 --2- 172.21.15.154:0/2145656990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 0x7f00ec102310 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:34.883 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.871+0000 7f00f0852640 1 -- 172.21.15.154:0/2145656990 >> 172.21.15.154:0/2145656990 conn(0x7f00ec0f7af0 msgr2=0x7f00ec0f9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:34.883 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.871+0000 7f00f0852640 1 -- 172.21.15.154:0/2145656990 shutdown_connections 2024-06-06T08:23:34.883 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.871+0000 7f00f0852640 1 -- 172.21.15.154:0/2145656990 wait complete. 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.871+0000 7f00f0852640 1 Processor -- start 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.872+0000 7f00f0852640 1 -- start start 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.872+0000 7f00f0852640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 0x7f00ec1125e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.872+0000 7f00f0852640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec112b20 con 0x7f00ec101f40 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.872+0000 7f00ea59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 0x7f00ec1125e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.872+0000 7f00ea59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 0x7f00ec1125e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:51136/0 (socket says 172.21.15.154:51136) 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.872+0000 7f00ea59c640 1 -- 172.21.15.154:0/4146524476 learned_addr learned my addr 172.21.15.154:0/4146524476 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.872+0000 7f00ea59c640 1 -- 172.21.15.154:0/4146524476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e40098b0 con 0x7f00ec101f40 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.873+0000 7f00ea59c640 1 --2- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 0x7f00ec1125e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009ba0 tx=0x7f00e4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.873+0000 7f00e2ffd640 1 -- 172.21.15.154:0/4146524476 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00e40328b0 con 0x7f00ec101f40 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.873+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ec112d20 con 0x7f00ec101f40 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.873+0000 7f00e2ffd640 1 -- 172.21.15.154:0/4146524476 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00e403d420 con 0x7f00ec101f40 2024-06-06T08:23:34.884 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.873+0000 7f00e2ffd640 1 -- 172.21.15.154:0/4146524476 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00e4031590 con 0x7f00ec101f40 2024-06-06T08:23:34.885 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.873+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec113170 con 0x7f00ec101f40 2024-06-06T08:23:34.885 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.874+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ec0fd6b0 con 0x7f00ec101f40 2024-06-06T08:23:34.885 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.874+0000 7f00e2ffd640 1 -- 172.21.15.154:0/4146524476 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7f00e402f030 con 0x7f00ec101f40 2024-06-06T08:23:34.885 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.874+0000 7f00e2ffd640 1 --2- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f00d80349f0 0x7f00d8036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:34.885 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.874+0000 7f00e2ffd640 1 -- 172.21.15.154:0/4146524476 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f00e4067cd0 con 0x7f00ec101f40 2024-06-06T08:23:34.885 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.874+0000 7f00e9d9b640 1 --2- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f00d80349f0 0x7f00d8036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:34.885 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.875+0000 7f00e9d9b640 1 --2- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f00d80349f0 0x7f00d8036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f00d4006fd0 tx=0x7f00d4006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:34.885 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:34.879+0000 7f00e2ffd640 1 -- 172.21.15.154:0/4146524476 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f00e404e530 con 0x7f00ec101f40 2024-06-06T08:23:35.113 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.113+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00ec10dc50 con 0x7f00ec101f40 2024-06-06T08:23:35.114 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.115+0000 7f00e2ffd640 1 -- 172.21.15.154:0/4146524476 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f00e403db30 con 0x7f00ec101f40 2024-06-06T08:23:35.114 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:35.114 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:35.115 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:35.116 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f00d80349f0 msgr2=0x7f00d8036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:35.116 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 --2- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f00d80349f0 0x7f00d8036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f00d4006fd0 tx=0x7f00d4006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:35.116 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 msgr2=0x7f00ec1125e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:35.117 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 --2- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 0x7f00ec1125e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009ba0 tx=0x7f00e4004d10 comp rx=0 tx=0).stop 2024-06-06T08:23:35.117 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 shutdown_connections 2024-06-06T08:23:35.117 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 --2- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f00d80349f0 0x7f00d8036ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:35.117 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 --2- 172.21.15.154:0/4146524476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec101f40 0x7f00ec1125e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:35.117 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 >> 172.21.15.154:0/4146524476 conn(0x7f00ec0f7af0 msgr2=0x7f00ec0f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:35.117 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 shutdown_connections 2024-06-06T08:23:35.117 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:35.117+0000 7f00f0852640 1 -- 172.21.15.154:0/4146524476 wait complete. 2024-06-06T08:23:36.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:36 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:36.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:36 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/4146524476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:36.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:36 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:36.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:36 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:36.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:36 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:36.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:36 smithi086 ceph-mon[29114]: Deploying daemon node-exporter.smithi154 on smithi154 2024-06-06T08:23:36.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:36.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:36.945 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T08:23:39.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.547+0000 7fbbe21e3640 1 -- 172.21.15.154:0/148089387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 msgr2=0x7fbbdc102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:39.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.547+0000 7fbbe21e3640 1 --2- 172.21.15.154:0/148089387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 0x7fbbdc102310 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc004850 tx=0x7fbbcc009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:39.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.547+0000 7fbbe21e3640 1 -- 172.21.15.154:0/148089387 shutdown_connections 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.547+0000 7fbbe21e3640 1 --2- 172.21.15.154:0/148089387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 0x7fbbdc102310 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.547+0000 7fbbe21e3640 1 -- 172.21.15.154:0/148089387 >> 172.21.15.154:0/148089387 conn(0x7fbbdc0f7af0 msgr2=0x7fbbdc0f9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.547+0000 7fbbe21e3640 1 -- 172.21.15.154:0/148089387 shutdown_connections 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.547+0000 7fbbe21e3640 1 -- 172.21.15.154:0/148089387 wait complete. 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.548+0000 7fbbe21e3640 1 Processor -- start 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.548+0000 7fbbe21e3640 1 -- start start 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.548+0000 7fbbe21e3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 0x7fbbdc1125e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.548+0000 7fbbe21e3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc112b20 con 0x7fbbdc101f40 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.548+0000 7fbbdb7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 0x7fbbdc1125e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:39.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.548+0000 7fbbdb7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 0x7fbbdc1125e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:38678/0 (socket says 172.21.15.154:38678) 2024-06-06T08:23:39.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.548+0000 7fbbdb7fe640 1 -- 172.21.15.154:0/1075446590 learned_addr learned my addr 172.21.15.154:0/1075446590 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:39.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.549+0000 7fbbdb7fe640 1 -- 172.21.15.154:0/1075446590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc005040 con 0x7fbbdc101f40 2024-06-06T08:23:39.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.549+0000 7fbbdb7fe640 1 --2- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 0x7fbbdc1125e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc004820 tx=0x7fbbcc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:39.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.549+0000 7fbbc3fff640 1 -- 172.21.15.154:0/1075446590 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbcc02cf10 con 0x7fbbdc101f40 2024-06-06T08:23:39.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.549+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbdc112d20 con 0x7fbbdc101f40 2024-06-06T08:23:39.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.549+0000 7fbbc3fff640 1 -- 172.21.15.154:0/1075446590 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbcc033b00 con 0x7fbbdc101f40 2024-06-06T08:23:39.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.549+0000 7fbbc3fff640 1 -- 172.21.15.154:0/1075446590 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbcc03db70 con 0x7fbbdc101f40 2024-06-06T08:23:39.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.549+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbdc113110 con 0x7fbbdc101f40 2024-06-06T08:23:39.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.550+0000 7fbbc3fff640 1 -- 172.21.15.154:0/1075446590 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7fbbcc030020 con 0x7fbbdc101f40 2024-06-06T08:23:39.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.550+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbdc0fd6b0 con 0x7fbbdc101f40 2024-06-06T08:23:39.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.550+0000 7fbbc3fff640 1 --2- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbbc40349f0 0x7fbbc4036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:39.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.550+0000 7fbbc3fff640 1 -- 172.21.15.154:0/1075446590 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fbbcc067000 con 0x7fbbdc101f40 2024-06-06T08:23:39.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.550+0000 7fbbdaffd640 1 --2- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbbc40349f0 0x7fbbc4036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:39.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.551+0000 7fbbdaffd640 1 --2- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbbc40349f0 0x7fbbc4036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0006fd0 tx=0x7fbbd0006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:39.556 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.555+0000 7fbbc3fff640 1 -- 172.21.15.154:0/1075446590 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbbcc045570 con 0x7fbbdc101f40 2024-06-06T08:23:39.798 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.799+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbdc10dc50 con 0x7fbbdc101f40 2024-06-06T08:23:39.798 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.799+0000 7fbbc3fff640 1 -- 172.21.15.154:0/1075446590 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbbcc045390 con 0x7fbbdc101f40 2024-06-06T08:23:39.799 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:39.799 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:39.799 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:39.800 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbbc40349f0 msgr2=0x7fbbc4036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:39.801 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 --2- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbbc40349f0 0x7fbbc4036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0006fd0 tx=0x7fbbd0006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:39.801 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 msgr2=0x7fbbdc1125e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:39.801 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 --2- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 0x7fbbdc1125e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc004820 tx=0x7fbbcc009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:39.801 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 shutdown_connections 2024-06-06T08:23:39.801 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 --2- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbbc40349f0 0x7fbbc4036ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:39.802 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 --2- 172.21.15.154:0/1075446590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbdc101f40 0x7fbbdc1125e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:39.802 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 >> 172.21.15.154:0/1075446590 conn(0x7fbbdc0f7af0 msgr2=0x7fbbdc0f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:39.802 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 shutdown_connections 2024-06-06T08:23:39.802 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:39.801+0000 7fbbe21e3640 1 -- 172.21.15.154:0/1075446590 wait complete. 2024-06-06T08:23:40.458 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:40.458 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:40.458 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:40.458 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:40.458 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi154.kxiapq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T08:23:40.458 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi154.kxiapq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-06T08:23:40.458 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T08:23:40.458 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:40.458 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: Deploying daemon mgr.smithi154.kxiapq on smithi154 2024-06-06T08:23:40.459 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:40.459 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:23:40.459 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:40 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/1075446590' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:41.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:41.575 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:41.869 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T08:23:43.737 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.710+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/1341859420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 msgr2=0x7f6bf4102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:43.737 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.710+0000 7f6bf9dd5640 1 --2- 172.21.15.154:0/1341859420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 0x7f6bf4102310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6be8004850 tx=0x7f6be8009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:43.737 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.710+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/1341859420 shutdown_connections 2024-06-06T08:23:43.737 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.710+0000 7f6bf9dd5640 1 --2- 172.21.15.154:0/1341859420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 0x7f6bf4102310 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:43.737 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.710+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/1341859420 >> 172.21.15.154:0/1341859420 conn(0x7f6bf4070e10 msgr2=0x7f6bf4071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:43.737 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.710+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/1341859420 shutdown_connections 2024-06-06T08:23:43.737 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.710+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/1341859420 wait complete. 2024-06-06T08:23:43.737 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.711+0000 7f6bf9dd5640 1 Processor -- start 2024-06-06T08:23:43.737 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.711+0000 7f6bf9dd5640 1 -- start start 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.711+0000 7f6bf9dd5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 0x7f6bf41125b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.711+0000 7f6bf9dd5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf4112af0 con 0x7f6bf4101f40 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.711+0000 7f6bf37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 0x7f6bf41125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.711+0000 7f6bf37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 0x7f6bf41125b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:38706/0 (socket says 172.21.15.154:38706) 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.711+0000 7f6bf37fe640 1 -- 172.21.15.154:0/3937383764 learned_addr learned my addr 172.21.15.154:0/3937383764 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.712+0000 7f6bf37fe640 1 -- 172.21.15.154:0/3937383764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8005040 con 0x7f6bf4101f40 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.712+0000 7f6bf37fe640 1 --2- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 0x7f6bf41125b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6be8004820 tx=0x7f6be8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.712+0000 7f6bdbfff640 1 -- 172.21.15.154:0/3937383764 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be802cf10 con 0x7f6bf4101f40 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.712+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf4112cf0 con 0x7f6bf4101f40 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.712+0000 7f6bdbfff640 1 -- 172.21.15.154:0/3937383764 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6be8033b00 con 0x7f6bf4101f40 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.712+0000 7f6bdbfff640 1 -- 172.21.15.154:0/3937383764 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be803db70 con 0x7f6bf4101f40 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.712+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf4113160 con 0x7f6bf4101f40 2024-06-06T08:23:43.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.713+0000 7f6bdbfff640 1 -- 172.21.15.154:0/3937383764 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7f6be8030020 con 0x7f6bf4101f40 2024-06-06T08:23:43.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.713+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf40fd6b0 con 0x7f6bf4101f40 2024-06-06T08:23:43.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.713+0000 7f6bdbfff640 1 --2- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6bdc0349f0 0x7f6bdc036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:43.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.713+0000 7f6bdbfff640 1 -- 172.21.15.154:0/3937383764 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6be8067e90 con 0x7f6bf4101f40 2024-06-06T08:23:43.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.714+0000 7f6bf2ffd640 1 --2- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6bdc0349f0 0x7f6bdc036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:43.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.714+0000 7f6bf2ffd640 1 --2- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6bdc0349f0 0x7f6bdc036ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6be4006fd0 tx=0x7f6be4006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:43.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.718+0000 7f6bdbfff640 1 -- 172.21.15.154:0/3937383764 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6be803b780 con 0x7f6bf4101f40 2024-06-06T08:23:43.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.959+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bf4106350 con 0x7f6bf4101f40 2024-06-06T08:23:43.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.960+0000 7f6bdbfff640 1 -- 172.21.15.154:0/3937383764 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6be803c3c0 con 0x7f6bf4101f40 2024-06-06T08:23:43.959 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:43.959 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:43.959 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:43.960 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6bdc0349f0 msgr2=0x7f6bdc036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:43.960 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 --2- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6bdc0349f0 0x7f6bdc036ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6be4006fd0 tx=0x7f6be4006db0 comp rx=0 tx=0).stop 2024-06-06T08:23:43.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 msgr2=0x7f6bf41125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:43.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 --2- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 0x7f6bf41125b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6be8004820 tx=0x7f6be8009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:43.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 shutdown_connections 2024-06-06T08:23:43.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 --2- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6bdc0349f0 0x7f6bdc036ea0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:43.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 --2- 172.21.15.154:0/3937383764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bf4101f40 0x7f6bf41125b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:43.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 >> 172.21.15.154:0/3937383764 conn(0x7f6bf4070e10 msgr2=0x7f6bf4103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:43.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 shutdown_connections 2024-06-06T08:23:43.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:43.962+0000 7f6bf9dd5640 1 -- 172.21.15.154:0/3937383764 wait complete. 2024-06-06T08:23:44.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:44.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:44.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:44.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:44.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T08:23:44.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:44.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:44 smithi086 ceph-mon[29114]: Deploying daemon mon.smithi154 on smithi154 2024-06-06T08:23:44.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:44 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/3937383764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:45.594 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:45.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:46.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:45 smithi086 ceph-mon[29114]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:46.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:45 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:46.525 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi154/config 2024-06-06T08:23:48.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:47 smithi086 ceph-mon[29114]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:48.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.141+0000 7fa76c4f2640 1 -- 172.21.15.154:0/1545251848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 msgr2=0x7fa764102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:48.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.141+0000 7fa76c4f2640 1 --2- 172.21.15.154:0/1545251848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 0x7fa764102420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa75c004850 tx=0x7fa75c009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:48.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.141+0000 7fa76c4f2640 1 -- 172.21.15.154:0/1545251848 shutdown_connections 2024-06-06T08:23:48.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.141+0000 7fa76c4f2640 1 --2- 172.21.15.154:0/1545251848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 0x7fa764102420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:48.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.141+0000 7fa76c4f2640 1 -- 172.21.15.154:0/1545251848 >> 172.21.15.154:0/1545251848 conn(0x7fa7640f7c20 msgr2=0x7fa7640fa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:48.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.142+0000 7fa76c4f2640 1 -- 172.21.15.154:0/1545251848 shutdown_connections 2024-06-06T08:23:48.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.142+0000 7fa76c4f2640 1 -- 172.21.15.154:0/1545251848 wait complete. 2024-06-06T08:23:48.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.142+0000 7fa76c4f2640 1 Processor -- start 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.142+0000 7fa76c4f2640 1 -- start start 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.142+0000 7fa76c4f2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 0x7fa764111620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.142+0000 7fa76c4f2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa764111b60 con 0x7fa764102050 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.142+0000 7fa76a28e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 0x7fa764111620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.142+0000 7fa76a28e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 0x7fa764111620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:57244/0 (socket says 172.21.15.154:57244) 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.142+0000 7fa76a28e640 1 -- 172.21.15.154:0/4227119285 learned_addr learned my addr 172.21.15.154:0/4227119285 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.143+0000 7fa76a28e640 1 -- 172.21.15.154:0/4227119285 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c005040 con 0x7fa764102050 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.143+0000 7fa76a28e640 1 --2- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 0x7fa764111620 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa75c004820 tx=0x7fa75c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.143+0000 7fa762ffd640 1 -- 172.21.15.154:0/4227119285 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa75c02cf10 con 0x7fa764102050 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.143+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa764111d00 con 0x7fa764102050 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.143+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7641120f0 con 0x7fa764102050 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.143+0000 7fa762ffd640 1 -- 172.21.15.154:0/4227119285 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa75c033b00 con 0x7fa764102050 2024-06-06T08:23:48.170 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.143+0000 7fa762ffd640 1 -- 172.21.15.154:0/4227119285 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa75c03db70 con 0x7fa764102050 2024-06-06T08:23:48.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.144+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7640fd7c0 con 0x7fa764102050 2024-06-06T08:23:48.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.144+0000 7fa762ffd640 1 -- 172.21.15.154:0/4227119285 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7fa75c030020 con 0x7fa764102050 2024-06-06T08:23:48.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.145+0000 7fa762ffd640 1 --2- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa74c0349f0 0x7fa74c036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:48.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.145+0000 7fa762ffd640 1 -- 172.21.15.154:0/4227119285 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa75c0671a0 con 0x7fa764102050 2024-06-06T08:23:48.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.145+0000 7fa769a8d640 1 --2- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa74c0349f0 0x7fa74c036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:48.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.145+0000 7fa769a8d640 1 --2- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa74c0349f0 0x7fa74c036ea0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa75800ad90 tx=0x7fa758009410 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:48.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.149+0000 7fa762ffd640 1 -- 172.21.15.154:0/4227119285 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa75c045570 con 0x7fa764102050 2024-06-06T08:23:48.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.393+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa76405eb10 con 0x7fa764102050 2024-06-06T08:23:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.393+0000 7fa762ffd640 1 -- 172.21.15.154:0/4227119285 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa75c045390 con 0x7fa764102050 2024-06-06T08:23:48.393 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:48.393 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:21:21.537411Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T08:23:48.393 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-06-06T08:23:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa74c0349f0 msgr2=0x7fa74c036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 --2- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa74c0349f0 0x7fa74c036ea0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa75800ad90 tx=0x7fa758009410 comp rx=0 tx=0).stop 2024-06-06T08:23:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 msgr2=0x7fa764111620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 --2- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 0x7fa764111620 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa75c004820 tx=0x7fa75c009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 shutdown_connections 2024-06-06T08:23:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 --2- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa74c0349f0 0x7fa74c036ea0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 --2- 172.21.15.154:0/4227119285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764102050 0x7fa764111620 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 >> 172.21.15.154:0/4227119285 conn(0x7fa7640f7c20 msgr2=0x7fa7640f9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 shutdown_connections 2024-06-06T08:23:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:48.396+0000 7fa76c4f2640 1 -- 172.21.15.154:0/4227119285 wait complete. 2024-06-06T08:23:49.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:48 smithi086 ceph-mon[29114]: Failed to load ceph-mgr modules: k8sevents 2024-06-06T08:23:49.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:48 smithi086 ceph-mon[29114]: Standby manager daemon smithi154.kxiapq started 2024-06-06T08:23:49.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:48 smithi086 ceph-mon[29114]: from='mgr.? 172.21.15.154:0/2265704313' entity='mgr.smithi154.kxiapq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi154.kxiapq/crt"}]: dispatch 2024-06-06T08:23:49.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:48 smithi086 ceph-mon[29114]: from='mgr.? 172.21.15.154:0/2265704313' entity='mgr.smithi154.kxiapq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-06T08:23:49.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:48 smithi086 ceph-mon[29114]: from='mgr.? 172.21.15.154:0/2265704313' entity='mgr.smithi154.kxiapq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi154.kxiapq/key"}]: dispatch 2024-06-06T08:23:49.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:48 smithi086 ceph-mon[29114]: from='mgr.? 172.21.15.154:0/2265704313' entity='mgr.smithi154.kxiapq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-06T08:23:49.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:48 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/4227119285' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:50.138 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T08:23:50.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T08:23:50.396 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi154/config 2024-06-06T08:23:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-06-06T08:23:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: mon.smithi086 calling monitor election 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: mon.smithi154 calling monitor election 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: mon.smithi086 is new leader, mons smithi086,smithi154 in quorum (ranks 0,1) 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: monmap e2: 2 mons at {smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],smithi154=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: fsmap 2024-06-06T08:23:54.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: osdmap e5: 0 total, 0 up, 0 in 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: mgrmap e18: smithi086.mfkgze(active, since 29s), standbys: smithi154.kxiapq 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: mgr module k8sevents crashed in daemon mgr.smithi086.mfkgze on host smithi086 at 2024-06-06T08:21:43.493542Z 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: mgr module k8sevents crashed in daemon mgr.smithi086.mfkgze on host smithi086 at 2024-06-06T08:21:52.990143Z 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: mgr module k8sevents crashed in daemon mgr.smithi086.mfkgze on host smithi086 at 2024-06-06T08:22:21.818401Z 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:54.087 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:54.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:53 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:54.821 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.820+0000 7ff3a7175640 1 -- 172.21.15.154:0/1579889258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3900034f0 msgr2=0x7ff390005970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.820+0000 7ff3a7175640 1 --2- 172.21.15.154:0/1579889258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3900034f0 0x7ff390005970 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0072f60 tx=0x7ff39400b040 comp rx=0 tx=0).stop 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.820+0000 7ff3a7175640 1 -- 172.21.15.154:0/1579889258 shutdown_connections 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.820+0000 7ff3a7175640 1 --2- 172.21.15.154:0/1579889258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3900034f0 0x7ff390005970 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.820+0000 7ff3a7175640 1 -- 172.21.15.154:0/1579889258 >> 172.21.15.154:0/1579889258 conn(0x7ff3a0070f20 msgr2=0x7ff3a0071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.820+0000 7ff3a7175640 1 -- 172.21.15.154:0/1579889258 shutdown_connections 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.820+0000 7ff3a7175640 1 -- 172.21.15.154:0/1579889258 wait complete. 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff3a7175640 1 Processor -- start 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff3a7175640 1 -- start start 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff3a7175640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3900034f0 0x7ff3a01127f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff3a7175640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 0x7ff3a010c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff3a7175640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a01132f0 con 0x7ff3900034f0 2024-06-06T08:23:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff3a7175640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0113430 con 0x7ff3a0112d30 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff39ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 0x7ff3a010c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff39ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 0x7ff3a010c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:48134/0 (socket says 172.21.15.154:48134) 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff39ffff640 1 -- 172.21.15.154:0/2770103676 learned_addr learned my addr 172.21.15.154:0/2770103676 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff39ffff640 1 -- 172.21.15.154:0/2770103676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 msgr2=0x7ff3a010c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff39ffff640 1 -- 172.21.15.154:0/2770103676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 msgr2=0x7ff3a010c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff39ffff640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 0x7ff3a010c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff3a4f11640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3900034f0 0x7ff3a01127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.821+0000 7ff39ffff640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 0x7ff3a010c790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.822+0000 7ff3a4f11640 1 -- 172.21.15.154:0/2770103676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 msgr2=0x7ff3a010c790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.822+0000 7ff3a4f11640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 0x7ff3a010c790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.822+0000 7ff3a4f11640 1 -- 172.21.15.154:0/2770103676 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff394005040 con 0x7ff3900034f0 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.822+0000 7ff3a4f11640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3900034f0 0x7ff3a01127f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff3940043d0 tx=0x7ff39400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.822+0000 7ff39d7fa640 1 -- 172.21.15.154:0/2770103676 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff394030b60 con 0x7ff3900034f0 2024-06-06T08:23:54.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.822+0000 7ff39d7fa640 1 -- 172.21.15.154:0/2770103676 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff39400f040 con 0x7ff3900034f0 2024-06-06T08:23:54.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.822+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a010cd90 con 0x7ff3900034f0 2024-06-06T08:23:54.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.822+0000 7ff39d7fa640 1 -- 172.21.15.154:0/2770103676 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff39403c650 con 0x7ff3900034f0 2024-06-06T08:23:54.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.822+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a010d2b0 con 0x7ff3900034f0 2024-06-06T08:23:54.825 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.823+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a00fd7c0 con 0x7ff3900034f0 2024-06-06T08:23:54.825 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.824+0000 7ff39d7fa640 1 -- 172.21.15.154:0/2770103676 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7ff39403e440 con 0x7ff3900034f0 2024-06-06T08:23:54.825 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.824+0000 7ff39d7fa640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff3840654f0 0x7ff3840679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:54.825 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.824+0000 7ff39d7fa640 1 -- 172.21.15.154:0/2770103676 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff3940a2960 con 0x7ff3900034f0 2024-06-06T08:23:54.825 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.825+0000 7ff39ffff640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff3840654f0 0x7ff3840679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:54.825 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.825+0000 7ff39ffff640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff3840654f0 0x7ff3840679a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff38c006fd0 tx=0x7ff38c008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:54.828 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:54.829+0000 7ff39d7fa640 1 -- 172.21.15.154:0/2770103676 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff394071b50 con 0x7ff3900034f0 2024-06-06T08:23:55.066 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.067+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3a004b480 con 0x7ff3900034f0 2024-06-06T08:23:55.067 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.068+0000 7ff39d7fa640 1 -- 172.21.15.154:0/2770103676 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7ff394042410 con 0x7ff3900034f0 2024-06-06T08:23:55.067 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-06-06T08:23:55.067 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":2,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","modified":"2024-06-06T08:23:48.742166Z","created":"2024-06-06T08:21:21.537411Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-06T08:23:55.067 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 2 2024-06-06T08:23:55.068 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:23:55.068 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:23:55.068 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:55.068 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:23:55.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff3840654f0 msgr2=0x7ff3840679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:55.069 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff3840654f0 0x7ff3840679a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff38c006fd0 tx=0x7ff38c008040 comp rx=0 tx=0).stop 2024-06-06T08:23:55.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3900034f0 msgr2=0x7ff3a01127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:55.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3900034f0 0x7ff3a01127f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff3940043d0 tx=0x7ff39400b040 comp rx=0 tx=0).stop 2024-06-06T08:23:55.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 shutdown_connections 2024-06-06T08:23:55.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a0112d30 0x7ff3a010c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:55.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff3840654f0 0x7ff3840679a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:55.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 --2- 172.21.15.154:0/2770103676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3900034f0 0x7ff3a01127f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:55.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 >> 172.21.15.154:0/2770103676 conn(0x7ff3a0070f20 msgr2=0x7ff3a0101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:55.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 shutdown_connections 2024-06-06T08:23:55.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:23:55.070+0000 7ff3a7175640 1 -- 172.21.15.154:0/2770103676 wait complete. 2024-06-06T08:23:55.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:23:55.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:23:55.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:55.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:23:55.973 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-06T08:23:55.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-06-06T08:23:56.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:55 smithi086 ceph-mon[29114]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:56.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:55 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/2770103676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:56.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:55 smithi154 ceph-mon[35428]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:56.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:55 smithi154 ceph-mon[35428]: from='client.? 172.21.15.154:0/2770103676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T08:23:56.282 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:23:57.333 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.333 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.333 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:57.333 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:23:57.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-06T08:23:57.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: Updating smithi154:/etc/ceph/ceph.conf 2024-06-06T08:23:57.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T08:23:57.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T08:23:57.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:57 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:57.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: Updating smithi154:/etc/ceph/ceph.conf 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T08:23:57.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:57 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.805+0000 7faeeecba640 1 -- 172.21.15.86:0/961696991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee80fcf80 msgr2=0x7faee80fd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.805+0000 7faeeecba640 1 --2- 172.21.15.86:0/961696991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee80fcf80 0x7faee80fd350 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faedc004850 tx=0x7faedc009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.805+0000 7faeeecba640 1 -- 172.21.15.86:0/961696991 shutdown_connections 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.805+0000 7faeeecba640 1 --2- 172.21.15.86:0/961696991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee80fcf80 0x7faee80fd350 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.805+0000 7faeeecba640 1 -- 172.21.15.86:0/961696991 >> 172.21.15.86:0/961696991 conn(0x7faee80f6a30 msgr2=0x7faee80f8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.805+0000 7faeeecba640 1 -- 172.21.15.86:0/961696991 shutdown_connections 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.805+0000 7faeeecba640 1 -- 172.21.15.86:0/961696991 wait complete. 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.806+0000 7faeeecba640 1 Processor -- start 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.806+0000 7faeeecba640 1 -- start start 2024-06-06T08:23:57.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.806+0000 7faeeecba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faee80fcf80 0x7faee81127b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.806+0000 7faeeecba640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee8112cf0 0x7faee810c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.806+0000 7faeeecba640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee81132b0 con 0x7faee8112cf0 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.806+0000 7faeeecba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee81133f0 con 0x7faee80fcf80 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee7fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee8112cf0 0x7faee810c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee7fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee8112cf0 0x7faee810c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52852/0 (socket says 172.21.15.86:52852) 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee7fff640 1 -- 172.21.15.86:0/4258339059 learned_addr learned my addr 172.21.15.86:0/4258339059 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee7fff640 1 -- 172.21.15.86:0/4258339059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faee80fcf80 msgr2=0x7faee81127b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee7fff640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faee80fcf80 0x7faee81127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee7fff640 1 -- 172.21.15.86:0/4258339059 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc005040 con 0x7faee8112cf0 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee7fff640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee8112cf0 0x7faee810c750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faed400b790 tx=0x7faed40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee57fa640 1 -- 172.21.15.86:0/4258339059 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faed400c910 con 0x7faee8112cf0 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee57fa640 1 -- 172.21.15.86:0/4258339059 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faed400be60 con 0x7faee8112cf0 2024-06-06T08:23:57.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee810cd50 con 0x7faee8112cf0 2024-06-06T08:23:57.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faee57fa640 1 -- 172.21.15.86:0/4258339059 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faed4014ea0 con 0x7faee8112cf0 2024-06-06T08:23:57.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.807+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee810d270 con 0x7faee8112cf0 2024-06-06T08:23:57.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.808+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee810d5a0 con 0x7faee8112cf0 2024-06-06T08:23:57.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.809+0000 7faee57fa640 1 -- 172.21.15.86:0/4258339059 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7faed4014440 con 0x7faee8112cf0 2024-06-06T08:23:57.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.809+0000 7faee57fa640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7faed80654f0 0x7faed80679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:23:57.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.809+0000 7faeeca56640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7faed80654f0 0x7faed80679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:23:57.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.809+0000 7faee57fa640 1 -- 172.21.15.86:0/4258339059 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7faed4080d40 con 0x7faee8112cf0 2024-06-06T08:23:57.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.810+0000 7faeeca56640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7faed80654f0 0x7faed80679a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faedc032410 tx=0x7faedc009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:23:57.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.813+0000 7faee57fa640 1 -- 172.21.15.86:0/4258339059 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7faed404cdf0 con 0x7faee8112cf0 2024-06-06T08:23:57.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.982+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7faee810deb0 con 0x7faee8112cf0 2024-06-06T08:23:57.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.983+0000 7faee57fa640 1 -- 172.21.15.86:0/4258339059 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7faed4019090 con 0x7faee8112cf0 2024-06-06T08:23:57.982 INFO:teuthology.orchestra.run.smithi086.stdout:# minimal ceph.conf for a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:23:57.982 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2024-06-06T08:23:57.982 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = a8550dee-23dd-11ef-bc9b-c7b262605968 2024-06-06T08:23:57.982 INFO:teuthology.orchestra.run.smithi086.stdout: mon_host = [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-06-06T08:23:57.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7faed80654f0 msgr2=0x7faed80679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:57.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7faed80654f0 0x7faed80679a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faedc032410 tx=0x7faedc009da0 comp rx=0 tx=0).stop 2024-06-06T08:23:57.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee8112cf0 msgr2=0x7faee810c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:23:57.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee8112cf0 0x7faee810c750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faed400b790 tx=0x7faed40091d0 comp rx=0 tx=0).stop 2024-06-06T08:23:57.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 shutdown_connections 2024-06-06T08:23:57.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7faed80654f0 0x7faed80679a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:57.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee8112cf0 0x7faee810c750 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:57.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 --2- 172.21.15.86:0/4258339059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faee80fcf80 0x7faee81127b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:23:57.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 >> 172.21.15.86:0/4258339059 conn(0x7faee80f6a30 msgr2=0x7faee80f8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:23:57.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 shutdown_connections 2024-06-06T08:23:57.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:23:57.985+0000 7faeeecba640 1 -- 172.21.15.86:0/4258339059 wait complete. 2024-06-06T08:23:58.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:58 smithi086 ceph-mon[29114]: Updating smithi154:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T08:23:58.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:58 smithi086 ceph-mon[29114]: Updating smithi086:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T08:23:58.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:58 smithi086 ceph-mon[29114]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:58.268 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:58 smithi086 ceph-mon[29114]: Reconfiguring mon.smithi086 (unknown last config time)... 2024-06-06T08:23:58.268 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:58 smithi086 ceph-mon[29114]: Reconfiguring daemon mon.smithi086 on smithi086 2024-06-06T08:23:58.268 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:23:58 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/4258339059' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:58.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:58 smithi154 ceph-mon[35428]: Updating smithi154:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T08:23:58.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:58 smithi154 ceph-mon[35428]: Updating smithi086:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T08:23:58.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:58 smithi154 ceph-mon[35428]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:23:58.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:58 smithi154 ceph-mon[35428]: Reconfiguring mon.smithi086 (unknown last config time)... 2024-06-06T08:23:58.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:58 smithi154 ceph-mon[35428]: Reconfiguring daemon mon.smithi086 on smithi086 2024-06-06T08:23:58.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:23:58 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/4258339059' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:23:58.566 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-06T08:23:58.566 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:23:58.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-06-06T08:23:58.594 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:23:58.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-06T08:23:58.663 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:23:58.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.conf 2024-06-06T08:23:58.695 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:23:58.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-06T08:23:58.764 INFO:tasks.cephadm:Deploying OSDs... 2024-06-06T08:23:58.765 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:23:58.765 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-06-06T08:23:58.780 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-06T08:23:58.781 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-06-06T08:23:58.836 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-06T08:23:58.837 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T08:23:58.837 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-06-06T08:23:58.837 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T08:23:58.837 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T08:23:58.837 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-06-06 08:23:18.564534294 +0000 2024-06-06T08:23:58.837 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-06-06 08:17:52.953906019 +0000 2024-06-06T08:23:58.837 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-06-06 08:17:52.953906019 +0000 2024-06-06T08:23:58.837 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-06-06 08:17:52.953906019 +0000 2024-06-06T08:23:58.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-06T08:23:58.900 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-06-06T08:23:58.900 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-06-06T08:23:58.900 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000250394 s, 2.0 MB/s 2024-06-06T08:23:58.901 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-06T08:23:58.957 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-06-06T08:23:59.013 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-06T08:23:59.013 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T08:23:59.013 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-06T08:23:59.013 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T08:23:59.013 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T08:23:59.013 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-06-06 08:23:18.565534274 +0000 2024-06-06T08:23:59.013 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-06-06 08:17:53.383896708 +0000 2024-06-06T08:23:59.013 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-06-06 08:17:53.383896708 +0000 2024-06-06T08:23:59.013 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-06-06 08:17:53.383896708 +0000 2024-06-06T08:23:59.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-06T08:23:59.076 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-06-06T08:23:59.076 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-06-06T08:23:59.076 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000180257 s, 2.8 MB/s 2024-06-06T08:23:59.077 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-06T08:23:59.132 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-06-06T08:23:59.188 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-06T08:23:59.188 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T08:23:59.188 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-06-06T08:23:59.188 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T08:23:59.188 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T08:23:59.188 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-06-06 08:23:18.565534274 +0000 2024-06-06T08:23:59.188 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-06-06 08:17:53.712889583 +0000 2024-06-06T08:23:59.188 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-06-06 08:17:53.712889583 +0000 2024-06-06T08:23:59.189 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-06-06 08:17:53.712889583 +0000 2024-06-06T08:23:59.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-06T08:23:59.251 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-06-06T08:23:59.252 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-06-06T08:23:59.252 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000204828 s, 2.5 MB/s 2024-06-06T08:23:59.253 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-06T08:23:59.309 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-06-06T08:23:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-06T08:23:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T08:23:59.364 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-06T08:23:59.364 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T08:23:59.365 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T08:23:59.365 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-06-06 08:23:18.565534274 +0000 2024-06-06T08:23:59.365 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-06-06 08:17:54.032882654 +0000 2024-06-06T08:23:59.365 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-06-06 08:17:54.032882654 +0000 2024-06-06T08:23:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-06-06 08:17:54.032882654 +0000 2024-06-06T08:23:59.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-06T08:23:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-06-06T08:23:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-06-06T08:23:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.00021933 s, 2.3 MB/s 2024-06-06T08:23:59.430 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-06T08:23:59.486 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:23:59.486 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-06-06T08:23:59.505 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-06T08:23:59.505 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-06-06T08:23:59.565 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-06T08:23:59.565 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T08:23:59.565 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-06T08:23:59.565 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T08:23:59.565 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T08:23:59.565 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-06-06 08:23:46.792931625 +0000 2024-06-06T08:23:59.565 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-06-06 08:17:52.961791804 +0000 2024-06-06T08:23:59.565 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-06-06 08:17:52.961791804 +0000 2024-06-06T08:23:59.565 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-06-06 08:17:52.961791804 +0000 2024-06-06T08:23:59.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-06T08:23:59.633 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-06-06T08:23:59.633 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-06-06T08:23:59.633 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000299297 s, 1.7 MB/s 2024-06-06T08:23:59.635 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-06T08:23:59.695 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-06-06T08:23:59.753 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-06T08:23:59.753 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T08:23:59.753 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-06T08:23:59.753 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T08:23:59.753 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T08:23:59.754 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-06-06 08:23:46.793931599 +0000 2024-06-06T08:23:59.754 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-06-06 08:17:53.290783368 +0000 2024-06-06T08:23:59.754 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-06-06 08:17:53.290783368 +0000 2024-06-06T08:23:59.754 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-06-06 08:17:53.290783368 +0000 2024-06-06T08:23:59.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-06T08:23:59.821 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-06-06T08:23:59.821 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-06-06T08:23:59.821 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000251835 s, 2.0 MB/s 2024-06-06T08:23:59.822 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-06T08:23:59.881 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-06-06T08:23:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-06T08:23:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T08:23:59.939 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-06-06T08:23:59.939 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T08:23:59.939 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T08:23:59.939 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-06-06 08:23:46.793931599 +0000 2024-06-06T08:23:59.939 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-06-06 08:17:53.626774752 +0000 2024-06-06T08:23:59.939 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-06-06 08:17:53.626774752 +0000 2024-06-06T08:23:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-06-06 08:17:53.626774752 +0000 2024-06-06T08:23:59.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-06T08:24:00.004 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-06-06T08:24:00.004 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-06-06T08:24:00.004 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000215047 s, 2.4 MB/s 2024-06-06T08:24:00.006 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-06T08:24:00.063 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-06-06T08:24:00.120 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-06T08:24:00.120 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T08:24:00.121 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-06-06T08:24:00.121 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T08:24:00.121 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T08:24:00.121 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-06-06 08:23:46.793931599 +0000 2024-06-06T08:24:00.121 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-06-06 08:17:53.962766137 +0000 2024-06-06T08:24:00.121 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-06-06 08:17:53.962766137 +0000 2024-06-06T08:24:00.121 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-06-06 08:17:53.962766137 +0000 2024-06-06T08:24:00.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-06T08:24:00.185 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-06-06T08:24:00.186 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-06-06T08:24:00.186 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.00024792 s, 2.1 MB/s 2024-06-06T08:24:00.187 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-06T08:24:00.244 INFO:tasks.cephadm:Deploying osd.0 on smithi086 with /dev/vg_nvme/lv_4... 2024-06-06T08:24:00.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-06T08:24:00.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:00 smithi086 ceph-mon[29114]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:00.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:00 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:00.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:00 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:00.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:00 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.mfkgze", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T08:24:00.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:00 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T08:24:00.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:00 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:00.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:00 smithi154 ceph-mon[35428]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:00.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:00 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:00.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:00 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:00.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:00 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.mfkgze", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T08:24:00.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:00 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T08:24:00.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:00 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:24:01.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:01 smithi086 ceph-mon[29114]: Reconfiguring mgr.smithi086.mfkgze (unknown last config time)... 2024-06-06T08:24:01.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:01 smithi086 ceph-mon[29114]: Reconfiguring daemon mgr.smithi086.mfkgze on smithi086 2024-06-06T08:24:01.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:01 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:01.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:01 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:01.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:01 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T08:24:01.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:01 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:01.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:01 smithi154 ceph-mon[35428]: Reconfiguring mgr.smithi086.mfkgze (unknown last config time)... 2024-06-06T08:24:01.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:01 smithi154 ceph-mon[35428]: Reconfiguring daemon mgr.smithi086.mfkgze on smithi086 2024-06-06T08:24:01.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:01 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:01.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:01 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:01.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:01 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T08:24:01.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:01 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:02.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:02 smithi086 ceph-mon[29114]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:02.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:02 smithi086 ceph-mon[29114]: Reconfiguring crash.smithi086 (monmap changed)... 2024-06-06T08:24:02.372 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:02 smithi086 ceph-mon[29114]: Reconfiguring daemon crash.smithi086 on smithi086 2024-06-06T08:24:02.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:02 smithi154 ceph-mon[35428]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:02.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:02 smithi154 ceph-mon[35428]: Reconfiguring crash.smithi086 (monmap changed)... 2024-06-06T08:24:02.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:02 smithi154 ceph-mon[35428]: Reconfiguring daemon crash.smithi086 on smithi086 2024-06-06T08:24:03.334 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-06T08:24:03.424 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-06T08:24:03.437 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-06-06T08:24:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-06-06T08:24:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011353 s, 924 MB/s 2024-06-06T08:24:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-06-06T08:24:04.191 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:04 smithi086 ceph-mon[29114]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:04.191 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:04 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:04.191 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:04 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:04.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-06-06T08:24:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:04 smithi154 ceph-mon[35428]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:04 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:04 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:05.004 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:24:05.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:05 smithi086 ceph-mon[29114]: Reconfiguring alertmanager.smithi086 (dependencies changed)... 2024-06-06T08:24:05.373 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:05 smithi086 ceph-mon[29114]: Reconfiguring daemon alertmanager.smithi086 on smithi086 2024-06-06T08:24:05.408 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:05 smithi154 ceph-mon[35428]: Reconfiguring alertmanager.smithi086 (dependencies changed)... 2024-06-06T08:24:05.408 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:05 smithi154 ceph-mon[35428]: Reconfiguring daemon alertmanager.smithi086 on smithi086 2024-06-06T08:24:06.295 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:06 smithi086 ceph-mon[29114]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:06.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:06 smithi154 ceph-mon[35428]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:08.115 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:08 smithi086 ceph-mon[29114]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:08.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:08 smithi154 ceph-mon[35428]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.743+0000 7f2a55be1640 1 -- 172.21.15.86:0/1400571295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fc290 msgr2=0x7f2a500fc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:24:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.743+0000 7f2a55be1640 1 --2- 172.21.15.86:0/1400571295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fc290 0x7f2a500fc6a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2a400048d0 tx=0x7f2a40009da0 comp rx=0 tx=0).stop 2024-06-06T08:24:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.743+0000 7f2a55be1640 1 -- 172.21.15.86:0/1400571295 shutdown_connections 2024-06-06T08:24:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.743+0000 7f2a55be1640 1 --2- 172.21.15.86:0/1400571295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a500fd560 0x7f2a500fd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.743+0000 7f2a55be1640 1 --2- 172.21.15.86:0/1400571295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fc290 0x7f2a500fc6a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.743+0000 7f2a55be1640 1 -- 172.21.15.86:0/1400571295 >> 172.21.15.86:0/1400571295 conn(0x7f2a500f7a70 msgr2=0x7f2a500f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.743+0000 7f2a55be1640 1 -- 172.21.15.86:0/1400571295 shutdown_connections 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.743+0000 7f2a55be1640 1 -- 172.21.15.86:0/1400571295 wait complete. 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.744+0000 7f2a55be1640 1 Processor -- start 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.744+0000 7f2a55be1640 1 -- start start 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.744+0000 7f2a55be1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a500fc290 0x7f2a5010cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.744+0000 7f2a55be1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fd560 0x7f2a5010d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.744+0000 7f2a55be1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5010e2a0 con 0x7f2a500fd560 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.744+0000 7f2a55be1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5010e410 con 0x7f2a500fc290 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.744+0000 7f2a4effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fd560 0x7f2a5010d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:24:08.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.744+0000 7f2a4effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fd560 0x7f2a5010d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58962/0 (socket says 172.21.15.86:58962) 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a4effd640 1 -- 172.21.15.86:0/2227188408 learned_addr learned my addr 172.21.15.86:0/2227188408 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a4f7fe640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a500fc290 0x7f2a5010cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a4effd640 1 -- 172.21.15.86:0/2227188408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a500fc290 msgr2=0x7f2a5010cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a4effd640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a500fc290 0x7f2a5010cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a4effd640 1 -- 172.21.15.86:0/2227188408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a40005040 con 0x7f2a500fd560 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a4effd640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fd560 0x7f2a5010d430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400b790 tx=0x7f2a440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a37fff640 1 -- 172.21.15.86:0/2227188408 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a4400c700 con 0x7f2a500fd560 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a37fff640 1 -- 172.21.15.86:0/2227188408 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a4400cd30 con 0x7f2a500fd560 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a37fff640 1 -- 172.21.15.86:0/2227188408 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a44013370 con 0x7f2a500fd560 2024-06-06T08:24:08.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a5010e6f0 con 0x7f2a500fd560 2024-06-06T08:24:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.745+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a5010ec10 con 0x7f2a500fd560 2024-06-06T08:24:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.746+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a500fb6f0 con 0x7f2a500fd560 2024-06-06T08:24:08.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.747+0000 7f2a37fff640 1 -- 172.21.15.86:0/2227188408 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7f2a44012070 con 0x7f2a500fd560 2024-06-06T08:24:08.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.747+0000 7f2a37fff640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2a380653a0 0x7f2a38067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:24:08.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.747+0000 7f2a37fff640 1 -- 172.21.15.86:0/2227188408 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f2a44080ef0 con 0x7f2a500fd560 2024-06-06T08:24:08.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.747+0000 7f2a4f7fe640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2a380653a0 0x7f2a38067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:24:08.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.748+0000 7f2a4f7fe640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2a380653a0 0x7f2a38067850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a40000c00 tx=0x7f2a40035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:24:08.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.751+0000 7f2a37fff640 1 -- 172.21.15.86:0/2227188408 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2a440501c0 con 0x7f2a500fd560 2024-06-06T08:24:08.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:08.934+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 --> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a5010eee0 con 0x7f2a380653a0 2024-06-06T08:24:09.316 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:24:09.316 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:24:09.317 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:09.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:24:09.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:24:09.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:10.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:10 smithi086 ceph-mon[29114]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:10.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:10 smithi086 ceph-mon[29114]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:24:10.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:10 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:24:10.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:10 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:10.235 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:10 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:10 smithi154 ceph-mon[35428]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:10 smithi154 ceph-mon[35428]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:24:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:10 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:24:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:10 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:10 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:11.401 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:11 smithi086 ceph-mon[29114]: Reconfiguring grafana.smithi086 (dependencies changed)... 2024-06-06T08:24:11.401 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:11 smithi086 ceph-mon[29114]: Reconfiguring daemon grafana.smithi086 on smithi086 2024-06-06T08:24:11.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:11 smithi154 ceph-mon[35428]: Reconfiguring grafana.smithi086 (dependencies changed)... 2024-06-06T08:24:11.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:11 smithi154 ceph-mon[35428]: Reconfiguring daemon grafana.smithi086 on smithi086 2024-06-06T08:24:12.252 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:12 smithi086 ceph-mon[29114]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:12.252 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:12 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1908361660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d6c551-bd2c-4826-a0e9-faf49f3759d8"}]: dispatch 2024-06-06T08:24:12.253 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:12 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1908361660' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d6c551-bd2c-4826-a0e9-faf49f3759d8"}]': finished 2024-06-06T08:24:12.253 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:12 smithi086 ceph-mon[29114]: osdmap e6: 1 total, 0 up, 1 in 2024-06-06T08:24:12.253 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:12.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:12 smithi154 ceph-mon[35428]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:12.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:12 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1908361660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50d6c551-bd2c-4826-a0e9-faf49f3759d8"}]: dispatch 2024-06-06T08:24:12.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:12 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1908361660' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50d6c551-bd2c-4826-a0e9-faf49f3759d8"}]': finished 2024-06-06T08:24:12.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:12 smithi154 ceph-mon[35428]: osdmap e6: 1 total, 0 up, 1 in 2024-06-06T08:24:12.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:13.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:13 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1171302857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:24:13.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:13 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1171302857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:24:14.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:14 smithi154 ceph-mon[35428]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:14.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:14 smithi086 ceph-mon[29114]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:16.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:16 smithi086 ceph-mon[29114]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:16.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:16 smithi154 ceph-mon[35428]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:18.305 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:18 smithi086 ceph-mon[29114]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:18.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:18 smithi154 ceph-mon[35428]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:20.252 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:20 smithi086 ceph-mon[29114]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:20.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:20 smithi154 ceph-mon[35428]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:22.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:22 smithi086 ceph-mon[29114]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:22.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:22 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:22.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:22 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:22.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:22 smithi086 ceph-mon[29114]: Reconfiguring prometheus.smithi086 (dependencies changed)... 2024-06-06T08:24:22.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:22 smithi086 ceph-mon[29114]: Reconfiguring daemon prometheus.smithi086 on smithi086 2024-06-06T08:24:22.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:22 smithi154 ceph-mon[35428]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:22.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:22 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:22.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:22 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:22.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:22 smithi154 ceph-mon[35428]: Reconfiguring prometheus.smithi086 (dependencies changed)... 2024-06-06T08:24:22.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:22 smithi154 ceph-mon[35428]: Reconfiguring daemon prometheus.smithi086 on smithi086 2024-06-06T08:24:23.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-06T08:24:23.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:23.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-06T08:24:23.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:24.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:24 smithi086 ceph-mon[29114]: Deploying daemon osd.0 on smithi086 2024-06-06T08:24:24.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:24 smithi086 ceph-mon[29114]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:24.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:24 smithi154 ceph-mon[35428]: Deploying daemon osd.0 on smithi086 2024-06-06T08:24:24.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:24 smithi154 ceph-mon[35428]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:25.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:25 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:24:25.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:25 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:24:26.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:26 smithi086 ceph-mon[29114]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:26.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:26 smithi154 ceph-mon[35428]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:27 smithi086 ceph-mon[29114]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:27 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:27 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:27 smithi086 ceph-mon[29114]: Reconfiguring crash.smithi154 (monmap changed)... 2024-06-06T08:24:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:27 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi154", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T08:24:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:27 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:27 smithi086 ceph-mon[29114]: Reconfiguring daemon crash.smithi154 on smithi154 2024-06-06T08:24:28.102 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:27 smithi154 ceph-mon[35428]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:28.102 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:27 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:28.102 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:27 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:28.102 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:27 smithi154 ceph-mon[35428]: Reconfiguring crash.smithi154 (monmap changed)... 2024-06-06T08:24:28.102 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:27 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi154", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T08:24:28.102 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:27 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:28.102 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:27 smithi154 ceph-mon[35428]: Reconfiguring daemon crash.smithi154 on smithi154 2024-06-06T08:24:29.144 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:28 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:29.145 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:28 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:29.145 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:28 smithi086 ceph-mon[29114]: Reconfiguring mgr.smithi154.kxiapq (monmap changed)... 2024-06-06T08:24:29.145 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:28 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi154.kxiapq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T08:24:29.145 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:28 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T08:24:29.145 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:28 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:29.145 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:28 smithi086 ceph-mon[29114]: Reconfiguring daemon mgr.smithi154.kxiapq on smithi154 2024-06-06T08:24:29.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:28 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:29.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:28 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:29.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:28 smithi154 ceph-mon[35428]: Reconfiguring mgr.smithi154.kxiapq (monmap changed)... 2024-06-06T08:24:29.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:28 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi154.kxiapq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T08:24:29.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:28 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T08:24:29.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:28 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:29.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:28 smithi154 ceph-mon[35428]: Reconfiguring daemon mgr.smithi154.kxiapq on smithi154 2024-06-06T08:24:30.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:30 smithi086 ceph-mon[29114]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:30.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:30.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:30.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:30 smithi086 ceph-mon[29114]: Reconfiguring mon.smithi154 (monmap changed)... 2024-06-06T08:24:30.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T08:24:30.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T08:24:30.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:30.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:30 smithi086 ceph-mon[29114]: Reconfiguring daemon mon.smithi154 on smithi154 2024-06-06T08:24:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:30 smithi154 ceph-mon[35428]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:30 smithi154 ceph-mon[35428]: Reconfiguring mon.smithi154 (monmap changed)... 2024-06-06T08:24:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T08:24:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T08:24:30.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:30.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:30 smithi154 ceph-mon[35428]: Reconfiguring daemon mon.smithi154 on smithi154 2024-06-06T08:24:31.552 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.552 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.552 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T08:24:31.552 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T08:24:31.552 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T08:24:31.552 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T08:24:31.552 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.552 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T08:24:31.552 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:24:31.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:31 smithi086 ceph-mon[29114]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:31.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T08:24:31.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T08:24:31.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T08:24:31.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:24:31.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:31 smithi154 ceph-mon[35428]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:32.819 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:32.819 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:32.819 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:32.819 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:32.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:32 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:32.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:32 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:32.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:32 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:32.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:32 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:34.578 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:34 smithi086 ceph-mon[29114]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:34.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:34 smithi154 ceph-mon[35428]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.698+0000 7f2a37fff640 1 -- 172.21.15.86:0/2227188408 <== mgr.14212 v2:172.21.15.86:6800/3471926911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2a5010eee0 con 0x7f2a380653a0 2024-06-06T08:24:35.699 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 0 on host 'smithi086' 2024-06-06T08:24:35.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2a380653a0 msgr2=0x7f2a38067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:24:35.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2a380653a0 0x7f2a38067850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a40000c00 tx=0x7f2a40035040 comp rx=0 tx=0).stop 2024-06-06T08:24:35.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fd560 msgr2=0x7f2a5010d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:24:35.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fd560 0x7f2a5010d430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400b790 tx=0x7f2a440091d0 comp rx=0 tx=0).stop 2024-06-06T08:24:35.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 shutdown_connections 2024-06-06T08:24:35.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2a380653a0 0x7f2a38067850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:24:35.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a500fd560 0x7f2a5010d430 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:24:35.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 --2- 172.21.15.86:0/2227188408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a500fc290 0x7f2a5010cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:24:35.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 >> 172.21.15.86:0/2227188408 conn(0x7f2a500f7a70 msgr2=0x7f2a500f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:24:35.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 shutdown_connections 2024-06-06T08:24:35.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:35.700+0000 7f2a55be1640 1 -- 172.21.15.86:0/2227188408 wait complete. 2024-06-06T08:24:36.244 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:36 smithi086 ceph-mon[29114]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:36.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:36 smithi086 ceph-mon[29114]: from='osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-06T08:24:36.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:36 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:36.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:36 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:36.269 DEBUG:teuthology.orchestra.run.smithi086:osd.0> sudo journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.0.service 2024-06-06T08:24:36.271 INFO:tasks.cephadm:Deploying osd.1 on smithi086 with /dev/vg_nvme/lv_3... 2024-06-06T08:24:36.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-06T08:24:36.603 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:36 smithi154 ceph-mon[35428]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:36.603 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:36 smithi154 ceph-mon[35428]: from='osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-06T08:24:36.603 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:36 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:36.603 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:36 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:24:37.568 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:37 smithi086 ceph-mon[29114]: from='osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-06T08:24:37.568 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:37 smithi086 ceph-mon[29114]: osdmap e7: 1 total, 0 up, 1 in 2024-06-06T08:24:37.568 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:37 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:37.568 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:37 smithi086 ceph-mon[29114]: from='osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-06T08:24:37.569 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:24:37 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0[47081]: 2024-06-06T08:24:37.232+0000 7f3fd26d1640 -1 osd.0 0 waiting for initial osdmap 2024-06-06T08:24:37.569 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:24:37 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0[47081]: 2024-06-06T08:24:37.237+0000 7f3fce50a640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T08:24:37.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:37 smithi154 ceph-mon[35428]: from='osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-06T08:24:37.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:37 smithi154 ceph-mon[35428]: osdmap e7: 1 total, 0 up, 1 in 2024-06-06T08:24:37.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:37 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:37.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:37 smithi154 ceph-mon[35428]: from='osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-06T08:24:38.353 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:38 smithi086 ceph-mon[29114]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:38.353 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:38 smithi086 ceph-mon[29114]: from='osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-06T08:24:38.353 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:38 smithi086 ceph-mon[29114]: osdmap e8: 1 total, 0 up, 1 in 2024-06-06T08:24:38.353 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:38.353 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:38.353 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:38.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:38 smithi154 ceph-mon[35428]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:38.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:38 smithi154 ceph-mon[35428]: from='osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-06T08:24:38.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:38 smithi154 ceph-mon[35428]: osdmap e8: 1 total, 0 up, 1 in 2024-06-06T08:24:38.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:38.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:38.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:39.468 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-06T08:24:39.469 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:39 smithi086 ceph-mon[29114]: purged_snaps scrub starts 2024-06-06T08:24:39.469 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:39 smithi086 ceph-mon[29114]: purged_snaps scrub ok 2024-06-06T08:24:39.469 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:39 smithi086 ceph-mon[29114]: osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] boot 2024-06-06T08:24:39.469 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:39 smithi086 ceph-mon[29114]: osdmap e9: 1 total, 1 up, 1 in 2024-06-06T08:24:39.469 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:39 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:39.469 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:39 smithi086 ceph-mon[29114]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:39.562 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-06T08:24:39.578 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-06-06T08:24:39.578 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-06-06T08:24:39.578 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128663 s, 815 MB/s 2024-06-06T08:24:39.578 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-06-06T08:24:39.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:39 smithi154 ceph-mon[35428]: purged_snaps scrub starts 2024-06-06T08:24:39.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:39 smithi154 ceph-mon[35428]: purged_snaps scrub ok 2024-06-06T08:24:39.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:39 smithi154 ceph-mon[35428]: osd.0 [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] boot 2024-06-06T08:24:39.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:39 smithi154 ceph-mon[35428]: osdmap e9: 1 total, 1 up, 1 in 2024-06-06T08:24:39.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:39 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T08:24:39.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:39 smithi154 ceph-mon[35428]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T08:24:40.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-06-06T08:24:40.459 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:40 smithi086 ceph-mon[29114]: osdmap e10: 1 total, 1 up, 1 in 2024-06-06T08:24:40.460 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:40.460 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:40.460 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:24:40.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:40 smithi154 ceph-mon[35428]: osdmap e10: 1 total, 1 up, 1 in 2024-06-06T08:24:40.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:40 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:40.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:40 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:40.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:40 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:24:41.028 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:24:41.487 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:41 smithi086 ceph-mon[29114]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:41.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:41 smithi154 ceph-mon[35428]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:43.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.532+0000 7fd8b01c2640 1 -- 172.21.15.86:0/3540237597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fbf80 msgr2=0x7fd8a80fa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:24:43.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.532+0000 7fd8b01c2640 1 --2- 172.21.15.86:0/3540237597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fbf80 0x7fd8a80fa3b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd8980048d0 tx=0x7fd898009da0 comp rx=0 tx=0).stop 2024-06-06T08:24:43.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.532+0000 7fd8b01c2640 1 -- 172.21.15.86:0/3540237597 shutdown_connections 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.532+0000 7fd8b01c2640 1 --2- 172.21.15.86:0/3540237597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fbf80 0x7fd8a80fa3b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.532+0000 7fd8b01c2640 1 --2- 172.21.15.86:0/3540237597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a80fb630 0x7fd8a80fba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.532+0000 7fd8b01c2640 1 -- 172.21.15.86:0/3540237597 >> 172.21.15.86:0/3540237597 conn(0x7fd8a80f6100 msgr2=0x7fd8a80f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.532+0000 7fd8b01c2640 1 -- 172.21.15.86:0/3540237597 shutdown_connections 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.532+0000 7fd8b01c2640 1 -- 172.21.15.86:0/3540237597 wait complete. 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8b01c2640 1 Processor -- start 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8b01c2640 1 -- start start 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8b01c2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fb630 0x7fd8a8113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8b01c2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a80fbf80 0x7fd8a8113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8b01c2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a81148b0 con 0x7fd8a80fb630 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8b01c2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a8114a20 con 0x7fd8a80fbf80 2024-06-06T08:24:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8adf5e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fb630 0x7fd8a8113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8adf5e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fb630 0x7fd8a8113500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51342/0 (socket says 172.21.15.86:51342) 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8adf5e640 1 -- 172.21.15.86:0/2374545490 learned_addr learned my addr 172.21.15.86:0/2374545490 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.533+0000 7fd8ad75d640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a80fbf80 0x7fd8a8113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.534+0000 7fd8adf5e640 1 -- 172.21.15.86:0/2374545490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a80fbf80 msgr2=0x7fd8a8113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.534+0000 7fd8adf5e640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a80fbf80 0x7fd8a8113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.534+0000 7fd8adf5e640 1 -- 172.21.15.86:0/2374545490 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd898005040 con 0x7fd8a80fb630 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.534+0000 7fd8adf5e640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fb630 0x7fd8a8113500 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400b790 tx=0x7fd8a40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.534+0000 7fd89e7fc640 1 -- 172.21.15.86:0/2374545490 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a400c700 con 0x7fd8a80fb630 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.534+0000 7fd89e7fc640 1 -- 172.21.15.86:0/2374545490 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8a400cd30 con 0x7fd8a80fb630 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.534+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a8114d00 con 0x7fd8a80fb630 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.534+0000 7fd89e7fc640 1 -- 172.21.15.86:0/2374545490 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a4013370 con 0x7fd8a80fb630 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.534+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a8115220 con 0x7fd8a80fb630 2024-06-06T08:24:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.535+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a810bc90 con 0x7fd8a80fb630 2024-06-06T08:24:43.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.536+0000 7fd89e7fc640 1 -- 172.21.15.86:0/2374545490 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7fd8a4012070 con 0x7fd8a80fb630 2024-06-06T08:24:43.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.536+0000 7fd89e7fc640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd8940653a0 0x7fd894067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:24:43.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.537+0000 7fd89e7fc640 1 -- 172.21.15.86:0/2374545490 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fd8a4081730 con 0x7fd8a80fb630 2024-06-06T08:24:43.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.537+0000 7fd8ad75d640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd8940653a0 0x7fd894067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:24:43.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.537+0000 7fd8ad75d640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd8940653a0 0x7fd894067850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd89802a530 tx=0x7fd898009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:24:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.540+0000 7fd89e7fc640 1 -- 172.21.15.86:0/2374545490 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd8a404ddf0 con 0x7fd8a80fb630 2024-06-06T08:24:43.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:24:43.719+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 --> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8a8115500 con 0x7fd8940653a0 2024-06-06T08:24:44.012 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:43 smithi086 ceph-mon[29114]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:44.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:43 smithi154 ceph-mon[35428]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:45.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:24:45.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:24:45.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:24:45.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:45.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: Detected new or changed devices on smithi086 2024-06-06T08:24:45.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:45.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:45.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:24:45.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: Adjusting osd_memory_target on smithi086 to 12864M 2024-06-06T08:24:45.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:45.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:45.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:24:45.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:45.086 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:44 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:24:45.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:24:45.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:24:45.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: Detected new or changed devices on smithi086 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: Adjusting osd_memory_target on smithi086 to 12864M 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:45.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:44 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:24:46.181 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:46 smithi086 ceph-mon[29114]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:46.182 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:46.182 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:46.359 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:46 smithi154 ceph-mon[35428]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:46.359 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:46.359 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:47.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:47 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3983318626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e6e7221-c7a2-48e9-b584-83b541395665"}]: dispatch 2024-06-06T08:24:47.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:47 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3983318626' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e6e7221-c7a2-48e9-b584-83b541395665"}]': finished 2024-06-06T08:24:47.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:47 smithi086 ceph-mon[29114]: osdmap e11: 2 total, 1 up, 2 in 2024-06-06T08:24:47.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:47 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:24:47.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:47 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2256386239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:24:47.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:47 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3983318626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e6e7221-c7a2-48e9-b584-83b541395665"}]: dispatch 2024-06-06T08:24:47.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:47 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3983318626' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e6e7221-c7a2-48e9-b584-83b541395665"}]': finished 2024-06-06T08:24:47.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:47 smithi154 ceph-mon[35428]: osdmap e11: 2 total, 1 up, 2 in 2024-06-06T08:24:47.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:47 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:24:47.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:47 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2256386239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:24:48.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:48 smithi154 ceph-mon[35428]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:48.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:48 smithi086 ceph-mon[29114]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:50.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:50 smithi086 ceph-mon[29114]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:24:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:50 smithi154 ceph-mon[35428]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:24:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:52.378 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:52 smithi086 ceph-mon[29114]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:52.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:52 smithi154 ceph-mon[35428]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:54.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:54 smithi086 ceph-mon[29114]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:54.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-06T08:24:54.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:54.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:54 smithi154 ceph-mon[35428]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:54.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-06T08:24:54.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:24:55.237 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:55 smithi086 ceph-mon[29114]: Deploying daemon osd.1 on smithi086 2024-06-06T08:24:55.237 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:55 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:24:55.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:55 smithi154 ceph-mon[35428]: Deploying daemon osd.1 on smithi086 2024-06-06T08:24:55.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:55 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:24:56.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:56 smithi086 ceph-mon[29114]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:56.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:56 smithi154 ceph-mon[35428]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:58.356 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:58 smithi086 ceph-mon[29114]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:58.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:58 smithi154 ceph-mon[35428]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:24:59.205 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:24:59.205 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:59.205 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:24:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:24:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:24:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:24:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:00.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:00 smithi086 ceph-mon[29114]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:25:00.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:00 smithi154 ceph-mon[35428]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:25:01.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:01 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:01.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:01 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:01.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:01 smithi086 ceph-mon[29114]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:25:01.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:01 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:01.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:01 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:01.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:01 smithi154 ceph-mon[35428]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:25:01.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.912+0000 7fd89e7fc640 1 -- 172.21.15.86:0/2374545490 <== mgr.14212 v2:172.21.15.86:6800/3471926911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd8a8115500 con 0x7fd8940653a0 2024-06-06T08:25:01.912 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 1 on host 'smithi086' 2024-06-06T08:25:01.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd8940653a0 msgr2=0x7fd894067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:01.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd8940653a0 0x7fd894067850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd89802a530 tx=0x7fd898009da0 comp rx=0 tx=0).stop 2024-06-06T08:25:01.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fb630 msgr2=0x7fd8a8113500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:01.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fb630 0x7fd8a8113500 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400b790 tx=0x7fd8a40091d0 comp rx=0 tx=0).stop 2024-06-06T08:25:01.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 shutdown_connections 2024-06-06T08:25:01.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a80fbf80 0x7fd8a8113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:01.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd8940653a0 0x7fd894067850 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:01.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 --2- 172.21.15.86:0/2374545490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a80fb630 0x7fd8a8113500 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:01.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 >> 172.21.15.86:0/2374545490 conn(0x7fd8a80f6100 msgr2=0x7fd8a80f77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:25:01.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 shutdown_connections 2024-06-06T08:25:01.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:01.915+0000 7fd8b01c2640 1 -- 172.21.15.86:0/2374545490 wait complete. 2024-06-06T08:25:02.496 DEBUG:teuthology.orchestra.run.smithi086:osd.1> sudo journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.1.service 2024-06-06T08:25:02.499 INFO:tasks.cephadm:Deploying osd.2 on smithi086 with /dev/vg_nvme/lv_2... 2024-06-06T08:25:02.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-06T08:25:03.028 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:25:03.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:02 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:03.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:02 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:03.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:02 smithi086 ceph-mon[29114]: from='osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-06T08:25:03.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:02 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:03.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:02 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:03.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:02 smithi154 ceph-mon[35428]: from='osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-06T08:25:04.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:03 smithi086 ceph-mon[29114]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:25:04.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:03 smithi086 ceph-mon[29114]: from='osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-06T08:25:04.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:03 smithi086 ceph-mon[29114]: osdmap e12: 2 total, 1 up, 2 in 2024-06-06T08:25:04.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:03 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:25:04.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:03 smithi086 ceph-mon[29114]: from='osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-06T08:25:04.085 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:03 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1[54687]: 2024-06-06T08:25:03.927+0000 7f04c04da640 -1 osd.1 0 waiting for initial osdmap 2024-06-06T08:25:04.086 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:03 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1[54687]: 2024-06-06T08:25:03.931+0000 7f04bc313640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T08:25:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:03 smithi154 ceph-mon[35428]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:25:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:03 smithi154 ceph-mon[35428]: from='osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-06T08:25:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:03 smithi154 ceph-mon[35428]: osdmap e12: 2 total, 1 up, 2 in 2024-06-06T08:25:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:03 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:25:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:03 smithi154 ceph-mon[35428]: from='osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-06T08:25:05.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:04 smithi086 ceph-mon[29114]: from='osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-06T08:25:05.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:04 smithi086 ceph-mon[29114]: osdmap e13: 2 total, 1 up, 2 in 2024-06-06T08:25:05.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:04 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:25:05.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:04 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:25:05.334 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:04 smithi154 ceph-mon[35428]: from='osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-06T08:25:05.334 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:04 smithi154 ceph-mon[35428]: osdmap e13: 2 total, 1 up, 2 in 2024-06-06T08:25:05.334 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:04 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:25:05.334 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:04 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:25:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-06T08:25:05.851 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-06T08:25:05.865 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-06-06T08:25:05.865 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-06-06T08:25:05.865 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012331 s, 850 MB/s 2024-06-06T08:25:05.865 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-06-06T08:25:05.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:05 smithi086 ceph-mon[29114]: purged_snaps scrub starts 2024-06-06T08:25:05.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:05 smithi086 ceph-mon[29114]: purged_snaps scrub ok 2024-06-06T08:25:05.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:05 smithi086 ceph-mon[29114]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:25:05.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:05 smithi086 ceph-mon[29114]: osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] boot 2024-06-06T08:25:05.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:05 smithi086 ceph-mon[29114]: osdmap e14: 2 total, 2 up, 2 in 2024-06-06T08:25:05.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:05 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:25:06.275 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:05 smithi154 ceph-mon[35428]: purged_snaps scrub starts 2024-06-06T08:25:06.275 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:05 smithi154 ceph-mon[35428]: purged_snaps scrub ok 2024-06-06T08:25:06.276 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:05 smithi154 ceph-mon[35428]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T08:25:06.276 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:05 smithi154 ceph-mon[35428]: osd.1 [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] boot 2024-06-06T08:25:06.276 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:05 smithi154 ceph-mon[35428]: osdmap e14: 2 total, 2 up, 2 in 2024-06-06T08:25:06.276 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:05 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T08:25:06.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-06-06T08:25:07.176 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:06 smithi086 ceph-mon[29114]: osdmap e15: 2 total, 2 up, 2 in 2024-06-06T08:25:07.406 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:25:07.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:06 smithi154 ceph-mon[35428]: osdmap e15: 2 total, 2 up, 2 in 2024-06-06T08:25:08.224 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:07 smithi086 ceph-mon[29114]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:08.224 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:07 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:08.224 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:07 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:08.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:07 smithi154 ceph-mon[35428]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:08.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:07 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:08.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:07 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:08.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.984+0000 7fcf53497640 1 -- 172.21.15.86:0/2152252633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 msgr2=0x7fcf4c0fb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:08.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.984+0000 7fcf53497640 1 --2- 172.21.15.86:0/2152252633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 0x7fcf4c0fb5a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009c50 tx=0x7fcf48004d10 comp rx=0 tx=0).stop 2024-06-06T08:25:08.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.984+0000 7fcf53497640 1 -- 172.21.15.86:0/2152252633 shutdown_connections 2024-06-06T08:25:08.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.984+0000 7fcf53497640 1 --2- 172.21.15.86:0/2152252633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c0fc460 0x7fcf4c0fc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.984+0000 7fcf53497640 1 --2- 172.21.15.86:0/2152252633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 0x7fcf4c0fb5a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.984+0000 7fcf53497640 1 -- 172.21.15.86:0/2152252633 >> 172.21.15.86:0/2152252633 conn(0x7fcf4c0f6970 msgr2=0x7fcf4c0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.984+0000 7fcf53497640 1 -- 172.21.15.86:0/2152252633 shutdown_connections 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.984+0000 7fcf53497640 1 -- 172.21.15.86:0/2152252633 wait complete. 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf53497640 1 Processor -- start 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf53497640 1 -- start start 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf53497640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 0x7fcf4c10be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf53497640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c0fc460 0x7fcf4c10c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf53497640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c10d150 con 0x7fcf4c0fb190 2024-06-06T08:25:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf53497640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c10d2c0 con 0x7fcf4c0fc460 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf51233640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 0x7fcf4c10be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf51233640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 0x7fcf4c10be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55048/0 (socket says 172.21.15.86:55048) 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf51233640 1 -- 172.21.15.86:0/3633248052 learned_addr learned my addr 172.21.15.86:0/3633248052 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf50a32640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c0fc460 0x7fcf4c10c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf51233640 1 -- 172.21.15.86:0/3633248052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c0fc460 msgr2=0x7fcf4c10c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf51233640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c0fc460 0x7fcf4c10c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf51233640 1 -- 172.21.15.86:0/3633248052 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf480098b0 con 0x7fcf4c0fb190 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.985+0000 7fcf50a32640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c0fc460 0x7fcf4c10c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.986+0000 7fcf51233640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 0x7fcf4c10be80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009c20 tx=0x7fcf48004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.986+0000 7fcf41ffb640 1 -- 172.21.15.86:0/3633248052 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf48032910 con 0x7fcf4c0fb190 2024-06-06T08:25:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.986+0000 7fcf41ffb640 1 -- 172.21.15.86:0/3633248052 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf48008ee0 con 0x7fcf4c0fb190 2024-06-06T08:25:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.986+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf4c10d540 con 0x7fcf4c0fb190 2024-06-06T08:25:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.986+0000 7fcf41ffb640 1 -- 172.21.15.86:0/3633248052 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf4803dcd0 con 0x7fcf4c0fb190 2024-06-06T08:25:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.986+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf4c10da60 con 0x7fcf4c0fb190 2024-06-06T08:25:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.987+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf4c104570 con 0x7fcf4c0fb190 2024-06-06T08:25:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.987+0000 7fcf41ffb640 1 -- 172.21.15.86:0/3633248052 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7fcf4802f030 con 0x7fcf4c0fb190 2024-06-06T08:25:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.988+0000 7fcf41ffb640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fcf380653a0 0x7fcf38067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:25:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.988+0000 7fcf41ffb640 1 -- 172.21.15.86:0/3633248052 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fcf480a3270 con 0x7fcf4c0fb190 2024-06-06T08:25:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.988+0000 7fcf50a32640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fcf380653a0 0x7fcf38067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:25:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.988+0000 7fcf50a32640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fcf380653a0 0x7fcf38067850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c006fd0 tx=0x7fcf3c008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:25:08.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:08.992+0000 7fcf41ffb640 1 -- 172.21.15.86:0/3633248052 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcf480720b0 con 0x7fcf4c0fb190 2024-06-06T08:25:09.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:09.176+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 --> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf4c10dbf0 con 0x7fcf380653a0 2024-06-06T08:25:10.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:09 smithi086 ceph-mon[29114]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:10.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:09 smithi086 ceph-mon[29114]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:25:10.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:25:10.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:25:10.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:10.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:25:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:09 smithi154 ceph-mon[35428]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:09 smithi154 ceph-mon[35428]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:25:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:25:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:25:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:10.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:25:12.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:11 smithi086 ceph-mon[29114]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:12.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:11 smithi154 ceph-mon[35428]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:13.150 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: Detected new or changed devices on smithi086 2024-06-06T08:25:13.150 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:13.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:13.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:13.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:13.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: Adjusting osd_memory_target on smithi086 to 6432M 2024-06-06T08:25:13.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:13.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:13.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:13.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:13.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:12 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:13.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: Detected new or changed devices on smithi086 2024-06-06T08:25:13.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:13.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:13.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:13.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:13.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: Adjusting osd_memory_target on smithi086 to 6432M 2024-06-06T08:25:13.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:13.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:13.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:13.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:13.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:12 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:14.253 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:13 smithi086 ceph-mon[29114]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:14.253 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:13 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3391506488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2ad1948-0ddb-4821-b150-f723c3877245"}]: dispatch 2024-06-06T08:25:14.253 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:13 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3391506488' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2ad1948-0ddb-4821-b150-f723c3877245"}]': finished 2024-06-06T08:25:14.253 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:13 smithi086 ceph-mon[29114]: osdmap e16: 3 total, 2 up, 3 in 2024-06-06T08:25:14.254 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:13 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:14.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:13 smithi154 ceph-mon[35428]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:14.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:13 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3391506488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2ad1948-0ddb-4821-b150-f723c3877245"}]: dispatch 2024-06-06T08:25:14.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:13 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3391506488' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2ad1948-0ddb-4821-b150-f723c3877245"}]': finished 2024-06-06T08:25:14.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:13 smithi154 ceph-mon[35428]: osdmap e16: 3 total, 2 up, 3 in 2024-06-06T08:25:14.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:13 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:15.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:14 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/4039821304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:25:15.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:14 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/4039821304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:25:16.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:15 smithi154 ceph-mon[35428]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:16.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:15 smithi086 ceph-mon[29114]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:17.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:17 smithi086 ceph-mon[29114]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:17.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:17.900 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:17.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:17.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:17.901 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:18.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:17 smithi154 ceph-mon[35428]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:18.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:18.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:18.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:18.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:18.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:19.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:19 smithi086 ceph-mon[29114]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:20.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:19 smithi154 ceph-mon[35428]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:20.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:20 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-06T08:25:20.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:20 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:21.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:20 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-06T08:25:21.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:20 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:22.018 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:21 smithi086 ceph-mon[29114]: Deploying daemon osd.2 on smithi086 2024-06-06T08:25:22.018 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:21 smithi086 ceph-mon[29114]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:22.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:21 smithi154 ceph-mon[35428]: Deploying daemon osd.2 on smithi086 2024-06-06T08:25:22.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:21 smithi154 ceph-mon[35428]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:24.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:23 smithi086 ceph-mon[29114]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:24.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:23 smithi154 ceph-mon[35428]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:25.000 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:25:25.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:24 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:25:26.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:25 smithi086 ceph-mon[29114]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:26.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:25 smithi154 ceph-mon[35428]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:26.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:26 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:26.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:26 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:26.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:26 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:26.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:26 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:26.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:26 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:26.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:26 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:28.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:27 smithi154 ceph-mon[35428]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:28.316 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:27 smithi086 ceph-mon[29114]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:29.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.960+0000 7fcf41ffb640 1 -- 172.21.15.86:0/3633248052 <== mgr.14212 v2:172.21.15.86:6800/3471926911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcf4c10dbf0 con 0x7fcf380653a0 2024-06-06T08:25:29.960 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 2 on host 'smithi086' 2024-06-06T08:25:29.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fcf380653a0 msgr2=0x7fcf38067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:29.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fcf380653a0 0x7fcf38067850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c006fd0 tx=0x7fcf3c008040 comp rx=0 tx=0).stop 2024-06-06T08:25:29.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 msgr2=0x7fcf4c10be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:29.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 0x7fcf4c10be80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009c20 tx=0x7fcf48004d10 comp rx=0 tx=0).stop 2024-06-06T08:25:29.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 shutdown_connections 2024-06-06T08:25:29.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c0fc460 0x7fcf4c10c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:29.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fcf380653a0 0x7fcf38067850 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:29.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 --2- 172.21.15.86:0/3633248052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf4c0fb190 0x7fcf4c10be80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:29.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 >> 172.21.15.86:0/3633248052 conn(0x7fcf4c0f6970 msgr2=0x7fcf4c0f8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:25:29.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 shutdown_connections 2024-06-06T08:25:29.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:25:29.962+0000 7fcf53497640 1 -- 172.21.15.86:0/3633248052 wait complete. 2024-06-06T08:25:30.260 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:29 smithi086 ceph-mon[29114]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:30.260 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:29 smithi086 ceph-mon[29114]: from='osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-06T08:25:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:29 smithi154 ceph-mon[35428]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:29 smithi154 ceph-mon[35428]: from='osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-06T08:25:30.579 DEBUG:teuthology.orchestra.run.smithi086:osd.2> sudo journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.2.service 2024-06-06T08:25:30.581 INFO:tasks.cephadm:Deploying osd.3 on smithi154 with /dev/vg_nvme/lv_4... 2024-06-06T08:25:30.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-06T08:25:30.841 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi154/config 2024-06-06T08:25:31.251 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: from='osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-06T08:25:31.251 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: osdmap e17: 3 total, 2 up, 3 in 2024-06-06T08:25:31.251 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:31.251 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: from='osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-06T08:25:31.251 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:31.251 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:31.251 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: from='osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-06T08:25:31.252 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: osdmap e18: 3 total, 2 up, 3 in 2024-06-06T08:25:31.252 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:31.252 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:30 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:31.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: from='osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-06T08:25:31.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: osdmap e17: 3 total, 2 up, 3 in 2024-06-06T08:25:31.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:31.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: from='osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-06T08:25:31.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:31.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:31.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: from='osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-06T08:25:31.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: osdmap e18: 3 total, 2 up, 3 in 2024-06-06T08:25:31.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:31.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:31.335 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2[61965]: 2024-06-06T08:25:30.917+0000 7f1bb8c10640 -1 osd.2 0 waiting for initial osdmap 2024-06-06T08:25:31.336 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:30 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2[61965]: 2024-06-06T08:25:30.922+0000 7f1bb524a640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T08:25:32.072 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:31 smithi154 ceph-mon[35428]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:32.072 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:32.073 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:31 smithi154 ceph-mon[35428]: osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] boot 2024-06-06T08:25:32.073 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:31 smithi154 ceph-mon[35428]: osdmap e19: 3 total, 3 up, 3 in 2024-06-06T08:25:32.073 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:31 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:32.277 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:31 smithi086 ceph-mon[29114]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:32.277 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:32.277 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:31 smithi086 ceph-mon[29114]: osd.2 [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] boot 2024-06-06T08:25:32.277 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:31 smithi086 ceph-mon[29114]: osdmap e19: 3 total, 3 up, 3 in 2024-06-06T08:25:32.277 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:31 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T08:25:32.488 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-06T08:25:32.571 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-06T08:25:32.585 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-06-06T08:25:32.585 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-06-06T08:25:32.585 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112555 s, 932 MB/s 2024-06-06T08:25:32.585 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-06-06T08:25:33.152 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_4 2024-06-06T08:25:33.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:32 smithi086 ceph-mon[29114]: purged_snaps scrub starts 2024-06-06T08:25:33.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:32 smithi086 ceph-mon[29114]: purged_snaps scrub ok 2024-06-06T08:25:33.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-06T08:25:33.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:33.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:32 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:33.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:32 smithi154 ceph-mon[35428]: purged_snaps scrub starts 2024-06-06T08:25:33.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:32 smithi154 ceph-mon[35428]: purged_snaps scrub ok 2024-06-06T08:25:33.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:32 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-06T08:25:33.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:32 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:33.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:32 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:33.418 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi154/config 2024-06-06T08:25:34.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:33 smithi154 ceph-mon[35428]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:34.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:33 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-06T08:25:34.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:33 smithi154 ceph-mon[35428]: osdmap e20: 3 total, 3 up, 3 in 2024-06-06T08:25:34.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:33 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-06T08:25:34.268 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:33 smithi086 ceph-mon[29114]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T08:25:34.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:33 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-06T08:25:34.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:33 smithi086 ceph-mon[29114]: osdmap e20: 3 total, 3 up, 3 in 2024-06-06T08:25:34.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:33 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-06T08:25:34.834 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:25:34 smithi086 sudo[63783]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T08:25:34.835 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:25:34 smithi086 sudo[63783]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:34.835 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:25:34 smithi086 sudo[63783]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:34.835 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:25:34 smithi086 sudo[63783]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:34.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.843+0000 7ff1f8c12640 1 -- 172.21.15.154:0/1783574736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fc4a0 msgr2=0x7ff1f40fc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.843+0000 7ff1f8c12640 1 --2- 172.21.15.154:0/1783574736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fc4a0 0x7ff1f40fc910 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80048d0 tx=0x7ff1e8009da0 comp rx=0 tx=0).stop 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 -- 172.21.15.154:0/1783574736 shutdown_connections 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 --2- 172.21.15.154:0/1783574736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fc4a0 0x7ff1f40fc910 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 --2- 172.21.15.154:0/1783574736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1f40fb1d0 0x7ff1f40fb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 -- 172.21.15.154:0/1783574736 >> 172.21.15.154:0/1783574736 conn(0x7ff1f40f6990 msgr2=0x7ff1f40f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 -- 172.21.15.154:0/1783574736 shutdown_connections 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 -- 172.21.15.154:0/1783574736 wait complete. 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 Processor -- start 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 -- start start 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fb1d0 0x7ff1f410be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:25:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1f40fc4a0 0x7ff1f410c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f410c9e0 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f8c12640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f410d120 con 0x7ff1f40fb1d0 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fb1d0 0x7ff1f410be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fb1d0 0x7ff1f410be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50016/0 (socket says 172.21.15.154:50016) 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f1d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1f40fc4a0 0x7ff1f410c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.844+0000 7ff1f259c640 1 -- 172.21.15.154:0/544931352 learned_addr learned my addr 172.21.15.154:0/544931352 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1f1d9b640 1 -- 172.21.15.154:0/544931352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fb1d0 msgr2=0x7ff1f410be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1f1d9b640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fb1d0 0x7ff1f410be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1f1d9b640 1 -- 172.21.15.154:0/544931352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e8005040 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1f259c640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fb1d0 0x7ff1f410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1f1d9b640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1f40fc4a0 0x7ff1f410c390 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80048a0 tx=0x7ff1e8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:25:34.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1e2ffd640 1 -- 172.21.15.154:0/544931352 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1e8038070 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.848 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f410d350 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.848 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1e2ffd640 1 -- 172.21.15.154:0/544931352 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1e80092b0 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.848 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1e2ffd640 1 -- 172.21.15.154:0/544931352 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1e8033680 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.848 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.845+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f410d860 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.848 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.846+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f40fa630 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.847+0000 7ff1e2ffd640 1 -- 172.21.15.154:0/544931352 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7ff1e8030020 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.847+0000 7ff1e2ffd640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff1dc0653a0 0x7ff1dc067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:25:34.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.847+0000 7ff1e2ffd640 1 -- 172.21.15.154:0/544931352 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3228+0+0 (secure 0 0 0) 0x7ff1e80a1b40 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.847+0000 7ff1f259c640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff1dc0653a0 0x7ff1dc067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:25:34.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.848+0000 7ff1f259c640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff1dc0653a0 0x7ff1dc067850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1f410bbe0 tx=0x7ff1e400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:25:34.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.851+0000 7ff1e2ffd640 1 -- 172.21.15.154:0/544931352 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff1e8070760 con 0x7ff1f40fc4a0 2024-06-06T08:25:34.983 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:34.984+0000 7ff1e2ffd640 1 -- 172.21.15.154:0/544931352 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff1e8070b30 con 0x7ff1f40fc4a0 2024-06-06T08:25:35.023 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:35.024+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 --> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1f410dcb0 con 0x7ff1dc0653a0 2024-06-06T08:25:35.024 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:34 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-06T08:25:35.024 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:34 smithi154 ceph-mon[35428]: osdmap e21: 3 total, 3 up, 3 in 2024-06-06T08:25:35.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:34 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-06T08:25:35.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:34 smithi086 ceph-mon[29114]: osdmap e21: 3 total, 3 up, 3 in 2024-06-06T08:25:35.229 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:25:34 smithi086 sudo[63793]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T08:25:35.229 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:25:34 smithi086 sudo[63793]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:35.229 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:25:34 smithi086 sudo[63793]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:35.230 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:25:34 smithi086 sudo[63793]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:35.494 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[64833]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T08:25:35.494 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[64833]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:35.494 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[64833]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:35.495 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[64833]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:35.750 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[66455]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T08:25:35.750 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[66455]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:35.750 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[66455]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:35.750 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[66455]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:36.014 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[66466]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T08:25:36.014 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[66466]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:36.014 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[66466]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:36.014 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:35 smithi086 sudo[66466]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:36.014 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:35 smithi086 ceph-mon[29114]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:36.014 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:35 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:36.014 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:35 smithi086 ceph-mon[29114]: mgrmap e19: smithi086.mfkgze(active, since 2m), standbys: smithi154.kxiapq 2024-06-06T08:25:36.014 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:35 smithi086 ceph-mon[29114]: osdmap e22: 3 total, 3 up, 3 in 2024-06-06T08:25:36.015 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:35 smithi086 ceph-mon[29114]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:25:36.015 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:35 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:25:36.015 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:35 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:25:36.015 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:35 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:36.134 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:35 smithi154 ceph-mon[35428]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:36.134 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:35 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:36.134 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:35 smithi154 ceph-mon[35428]: mgrmap e19: smithi086.mfkgze(active, since 2m), standbys: smithi154.kxiapq 2024-06-06T08:25:36.135 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:35 smithi154 ceph-mon[35428]: osdmap e22: 3 total, 3 up, 3 in 2024-06-06T08:25:36.135 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:35 smithi154 ceph-mon[35428]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:25:36.135 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:35 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:25:36.135 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:35 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:25:36.135 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:35 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:36.265 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66473]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T08:25:36.265 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66473]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:36.265 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66473]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:36.265 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66473]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:36.524 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66476]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T08:25:36.524 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66476]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:36.524 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66476]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:36.524 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66476]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:36.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66480]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T08:25:36.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66480]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:36.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66480]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:36.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:36 smithi086 sudo[66480]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:37.045 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:36 smithi154 sudo[38643]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T08:25:37.045 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:36 smithi154 sudo[38643]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:37.045 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:36 smithi154 sudo[38643]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:37.045 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:36 smithi154 sudo[38643]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:37.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:37 smithi086 ceph-mon[29114]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T08:25:37.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:37 smithi086 ceph-mon[29114]: Detected new or changed devices on smithi086 2024-06-06T08:25:37.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:37 smithi086 ceph-mon[29114]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T08:25:37.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:37 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-06-06T08:25:37.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:37 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:25:37.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:37 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:37.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:37 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-06-06T08:25:37.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:37 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:25:37.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 sudo[38652]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T08:25:37.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 sudo[38652]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T08:25:37.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 sudo[38652]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T08:25:37.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 sudo[38652]: pam_unix(sudo:session): session closed for user root 2024-06-06T08:25:37.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 ceph-mon[35428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T08:25:37.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 ceph-mon[35428]: Detected new or changed devices on smithi086 2024-06-06T08:25:37.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 ceph-mon[35428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T08:25:37.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-06-06T08:25:37.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:25:37.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:37.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-06-06T08:25:37.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:37 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-06-06T08:25:38.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T08:25:38.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:38.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T08:25:38.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:38.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:38.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:38.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:38.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: Adjusting osd_memory_target on smithi086 to 4288M 2024-06-06T08:25:38.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/2503077604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "197f72f4-3b9e-4f36-a8fe-854cc214c6ee"}]: dispatch 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "197f72f4-3b9e-4f36-a8fe-854cc214c6ee"}]: dispatch 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "197f72f4-3b9e-4f36-a8fe-854cc214c6ee"}]': finished 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: osdmap e23: 4 total, 3 up, 4 in 2024-06-06T08:25:38.336 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:38.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T08:25:38.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: Adjusting osd_memory_target on smithi086 to 4288M 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:38.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:38.419 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='client.? 172.21.15.154:0/2503077604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "197f72f4-3b9e-4f36-a8fe-854cc214c6ee"}]: dispatch 2024-06-06T08:25:38.419 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "197f72f4-3b9e-4f36-a8fe-854cc214c6ee"}]: dispatch 2024-06-06T08:25:38.419 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "197f72f4-3b9e-4f36-a8fe-854cc214c6ee"}]': finished 2024-06-06T08:25:38.419 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: osdmap e23: 4 total, 3 up, 4 in 2024-06-06T08:25:38.419 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:39.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:39 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/2279650096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:25:39.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:39 smithi154 ceph-mon[35428]: from='client.? 172.21.15.154:0/2279650096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:25:40.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:40 smithi086 ceph-mon[29114]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:40.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:25:40.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:40 smithi154 ceph-mon[35428]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:40.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:40 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:25:42.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:42 smithi154 ceph-mon[35428]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:42.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:42 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:42.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:42 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:42.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:42 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:42.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:42 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:42.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:42 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:42.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:42 smithi086 ceph-mon[29114]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:42.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:42 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:42.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:42 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:42.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:42 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:42.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:42 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:42.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:42 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:44.165 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:44 smithi154 ceph-mon[35428]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-06-06T08:25:44.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:44 smithi086 ceph-mon[29114]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-06-06T08:25:45.131 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:45 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-06T08:25:45.131 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:45 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:45.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:45 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-06T08:25:45.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:45 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:46.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:46 smithi086 ceph-mon[29114]: Deploying daemon osd.3 on smithi154 2024-06-06T08:25:46.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:46 smithi086 ceph-mon[29114]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-06-06T08:25:46.338 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:46 smithi154 ceph-mon[35428]: Deploying daemon osd.3 on smithi154 2024-06-06T08:25:46.338 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:46 smithi154 ceph-mon[35428]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-06-06T08:25:48.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:48 smithi086 ceph-mon[29114]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-06-06T08:25:48.364 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:48 smithi154 ceph-mon[35428]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-06-06T08:25:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:50 smithi154 ceph-mon[35428]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-06-06T08:25:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:50 smithi086 ceph-mon[29114]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-06-06T08:25:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:52.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:52 smithi086 ceph-mon[29114]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-06T08:25:52.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:52 smithi154 ceph-mon[35428]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-06T08:25:52.713 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 3 on host 'smithi154' 2024-06-06T08:25:52.714 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.713+0000 7ff1e2ffd640 1 -- 172.21.15.154:0/544931352 <== mgr.14212 v2:172.21.15.86:6800/3471926911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff1f410dcb0 con 0x7ff1dc0653a0 2024-06-06T08:25:52.714 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.715+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff1dc0653a0 msgr2=0x7ff1dc067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.715+0000 7ff1f8c12640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff1dc0653a0 0x7ff1dc067850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1f410bbe0 tx=0x7ff1e400b040 comp rx=0 tx=0).stop 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.715+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1f40fc4a0 msgr2=0x7ff1f410c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.715+0000 7ff1f8c12640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1f40fc4a0 0x7ff1f410c390 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80048a0 tx=0x7ff1e8009da0 comp rx=0 tx=0).stop 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.715+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 shutdown_connections 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.715+0000 7ff1f8c12640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff1dc0653a0 0x7ff1dc067850 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.715+0000 7ff1f8c12640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1f40fc4a0 0x7ff1f410c390 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.715+0000 7ff1f8c12640 1 --2- 172.21.15.154:0/544931352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f40fb1d0 0x7ff1f410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.715+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 >> 172.21.15.154:0/544931352 conn(0x7ff1f40f6990 msgr2=0x7ff1f40f8050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.716+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 shutdown_connections 2024-06-06T08:25:52.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:52.716+0000 7ff1f8c12640 1 -- 172.21.15.154:0/544931352 wait complete. 2024-06-06T08:25:53.298 DEBUG:teuthology.orchestra.run.smithi154:osd.3> sudo journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.3.service 2024-06-06T08:25:53.300 INFO:tasks.cephadm:Deploying osd.4 on smithi154 with /dev/vg_nvme/lv_3... 2024-06-06T08:25:53.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-06T08:25:53.627 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi154/config 2024-06-06T08:25:53.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:53 smithi154 ceph-mon[35428]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-06T08:25:53.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:53 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:53.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:53 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:53.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:53 smithi154 ceph-mon[35428]: from='osd.3 [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T08:25:53.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:53 smithi154 ceph-mon[35428]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T08:25:53.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:53 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:53.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:53 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:53 smithi086 ceph-mon[29114]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-06T08:25:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:53 smithi086 ceph-mon[29114]: from='osd.3 [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T08:25:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:53 smithi086 ceph-mon[29114]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T08:25:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:54.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:53 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:54.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:54 smithi154 ceph-mon[35428]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-06T08:25:54.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:54 smithi154 ceph-mon[35428]: osdmap e24: 4 total, 3 up, 4 in 2024-06-06T08:25:54.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:54.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:54 smithi154 ceph-mon[35428]: from='osd.3 [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:25:54.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:54 smithi154 ceph-mon[35428]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:25:54.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:25:54.918 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:25:54 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3[40879]: 2024-06-06T08:25:54.721+0000 7fc9a0027640 -1 osd.3 0 waiting for initial osdmap 2024-06-06T08:25:54.918 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:25:54 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3[40879]: 2024-06-06T08:25:54.726+0000 7fc99ce62640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T08:25:55.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:54 smithi086 ceph-mon[29114]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-06T08:25:55.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:54 smithi086 ceph-mon[29114]: osdmap e24: 4 total, 3 up, 4 in 2024-06-06T08:25:55.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:55.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:54 smithi086 ceph-mon[29114]: from='osd.3 [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:25:55.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:54 smithi086 ceph-mon[29114]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:25:55.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:25:55.352 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-06T08:25:55.442 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-06T08:25:55.457 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-06-06T08:25:55.457 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-06-06T08:25:55.457 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127201 s, 824 MB/s 2024-06-06T08:25:55.457 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-06-06T08:25:55.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:55 smithi154 ceph-mon[35428]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:55.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:55 smithi154 ceph-mon[35428]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-06-06T08:25:55.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:55 smithi154 ceph-mon[35428]: osdmap e25: 4 total, 3 up, 4 in 2024-06-06T08:25:55.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:55 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:55.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:55 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:55.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:55 smithi154 ceph-mon[35428]: osd.3 [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] boot 2024-06-06T08:25:55.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:55 smithi154 ceph-mon[35428]: osdmap e26: 4 total, 4 up, 4 in 2024-06-06T08:25:55.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:55 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:56.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:55 smithi086 ceph-mon[29114]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-06T08:25:56.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:55 smithi086 ceph-mon[29114]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-06-06T08:25:56.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:55 smithi086 ceph-mon[29114]: osdmap e25: 4 total, 3 up, 4 in 2024-06-06T08:25:56.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:55 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:56.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:55 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:56.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:55 smithi086 ceph-mon[29114]: osd.3 [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] boot 2024-06-06T08:25:56.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:55 smithi086 ceph-mon[29114]: osdmap e26: 4 total, 4 up, 4 in 2024-06-06T08:25:56.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:55 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T08:25:56.814 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:56 smithi154 ceph-mon[35428]: purged_snaps scrub starts 2024-06-06T08:25:56.815 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:56 smithi154 ceph-mon[35428]: purged_snaps scrub ok 2024-06-06T08:25:56.815 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:56 smithi154 ceph-mon[35428]: osdmap e27: 4 total, 4 up, 4 in 2024-06-06T08:25:56.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_3 2024-06-06T08:25:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:56 smithi086 ceph-mon[29114]: purged_snaps scrub starts 2024-06-06T08:25:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:56 smithi086 ceph-mon[29114]: purged_snaps scrub ok 2024-06-06T08:25:57.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:56 smithi086 ceph-mon[29114]: osdmap e27: 4 total, 4 up, 4 in 2024-06-06T08:25:57.275 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi154/config 2024-06-06T08:25:58.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:57 smithi086 ceph-mon[29114]: pgmap v91: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:25:58.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:57 smithi086 ceph-mon[29114]: osdmap e28: 4 total, 4 up, 4 in 2024-06-06T08:25:58.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:57 smithi154 ceph-mon[35428]: pgmap v91: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:25:58.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:57 smithi154 ceph-mon[35428]: osdmap e28: 4 total, 4 up, 4 in 2024-06-06T08:25:59.355 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.344+0000 7fe04af1b640 1 -- 172.21.15.154:0/2124549736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fc290 msgr2=0x7fe0440fc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:59.355 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.344+0000 7fe04af1b640 1 --2- 172.21.15.154:0/2124549736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fc290 0x7fe0440fc6a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe0380048d0 tx=0x7fe038009da0 comp rx=0 tx=0).stop 2024-06-06T08:25:59.355 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.344+0000 7fe04af1b640 1 -- 172.21.15.154:0/2124549736 shutdown_connections 2024-06-06T08:25:59.355 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.344+0000 7fe04af1b640 1 --2- 172.21.15.154:0/2124549736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0440fd560 0x7fe0440fd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:59.355 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.344+0000 7fe04af1b640 1 --2- 172.21.15.154:0/2124549736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fc290 0x7fe0440fc6a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:59.355 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.344+0000 7fe04af1b640 1 -- 172.21.15.154:0/2124549736 >> 172.21.15.154:0/2124549736 conn(0x7fe0440f7a70 msgr2=0x7fe0440f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:25:59.355 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.344+0000 7fe04af1b640 1 -- 172.21.15.154:0/2124549736 shutdown_connections 2024-06-06T08:25:59.355 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.344+0000 7fe04af1b640 1 -- 172.21.15.154:0/2124549736 wait complete. 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe04af1b640 1 Processor -- start 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe04af1b640 1 -- start start 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe04af1b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0440fc290 0x7fe04410cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe04af1b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fd560 0x7fe04410d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe04af1b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04410e2d0 con 0x7fe0440fc290 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe04af1b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04410e440 con 0x7fe0440fd560 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe043fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fd560 0x7fe04410d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe043fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fd560 0x7fe04410d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:45468/0 (socket says 172.21.15.154:45468) 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe043fff640 1 -- 172.21.15.154:0/2745433594 learned_addr learned my addr 172.21.15.154:0/2745433594 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe043fff640 1 -- 172.21.15.154:0/2745433594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0440fc290 msgr2=0x7fe04410cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:25:59.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe043fff640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0440fc290 0x7fe04410cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:25:59.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe043fff640 1 -- 172.21.15.154:0/2745433594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe038005040 con 0x7fe0440fd560 2024-06-06T08:25:59.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.345+0000 7fe043fff640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fd560 0x7fe04410d460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe03000b790 tx=0x7fe0300091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:25:59.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.346+0000 7fe0417fa640 1 -- 172.21.15.154:0/2745433594 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe03000c700 con 0x7fe0440fd560 2024-06-06T08:25:59.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.346+0000 7fe0417fa640 1 -- 172.21.15.154:0/2745433594 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe03000cd30 con 0x7fe0440fd560 2024-06-06T08:25:59.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.346+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04410e720 con 0x7fe0440fd560 2024-06-06T08:25:59.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.346+0000 7fe0417fa640 1 -- 172.21.15.154:0/2745433594 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe030013370 con 0x7fe0440fd560 2024-06-06T08:25:59.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.346+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04410ec40 con 0x7fe0440fd560 2024-06-06T08:25:59.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.347+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0440fb6f0 con 0x7fe0440fd560 2024-06-06T08:25:59.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.347+0000 7fe0417fa640 1 -- 172.21.15.154:0/2745433594 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe030012070 con 0x7fe0440fd560 2024-06-06T08:25:59.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.347+0000 7fe0417fa640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe034065430 0x7fe0340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:25:59.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.348+0000 7fe0417fa640 1 -- 172.21.15.154:0/2745433594 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fe030081db0 con 0x7fe0440fd560 2024-06-06T08:25:59.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.348+0000 7fe048cb7640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe034065430 0x7fe0340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:25:59.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.348+0000 7fe048cb7640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe034065430 0x7fe0340678e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe038009f80 tx=0x7fe038009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:25:59.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.352+0000 7fe0417fa640 1 -- 172.21.15.154:0/2745433594 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe030050630 con 0x7fe0440fd560 2024-06-06T08:25:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: osdmap e29: 4 total, 4 up, 4 in 2024-06-06T08:25:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:25:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: Detected new or changed devices on smithi154 2024-06-06T08:25:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: Adjusting osd_memory_target on smithi154 to 15936M 2024-06-06T08:25:59.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:59.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:59.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:59.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:59.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:59.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:59.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:59.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:25:59 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:59.522 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:25:59.523+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 --> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe04410eec0 con 0x7fe034065430 2024-06-06T08:25:59.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: osdmap e29: 4 total, 4 up, 4 in 2024-06-06T08:25:59.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:25:59.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: Detected new or changed devices on smithi154 2024-06-06T08:25:59.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:59.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:59.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:25:59.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: Adjusting osd_memory_target on smithi154 to 15936M 2024-06-06T08:25:59.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:59.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:59.586 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:59.586 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:25:59.586 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:25:59.586 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:25:59.586 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:25:59.586 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:25:59 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:00.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:00 smithi086 ceph-mon[29114]: osdmap e30: 4 total, 4 up, 4 in 2024-06-06T08:26:00.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:00 smithi086 ceph-mon[29114]: from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:26:00.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:00 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:26:00.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:00 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:26:00.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:00 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:00.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:00 smithi154 ceph-mon[35428]: osdmap e30: 4 total, 4 up, 4 in 2024-06-06T08:26:00.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:00 smithi154 ceph-mon[35428]: from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:26:00.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:00 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:26:00.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:00 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:26:00.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:00 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:01.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:01 smithi154 ceph-mon[35428]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:01.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:01 smithi086 ceph-mon[29114]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:02.404 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:02 smithi154 ceph-mon[35428]: from='client.? 172.21.15.154:0/2633130713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f63fff7-ffe7-457f-a2c0-a5799c83ba32"}]: dispatch 2024-06-06T08:26:02.404 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:02 smithi154 ceph-mon[35428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f63fff7-ffe7-457f-a2c0-a5799c83ba32"}]: dispatch 2024-06-06T08:26:02.404 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:02 smithi154 ceph-mon[35428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f63fff7-ffe7-457f-a2c0-a5799c83ba32"}]': finished 2024-06-06T08:26:02.405 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:02 smithi154 ceph-mon[35428]: osdmap e31: 5 total, 4 up, 5 in 2024-06-06T08:26:02.405 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:02 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:02.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:02 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/2633130713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f63fff7-ffe7-457f-a2c0-a5799c83ba32"}]: dispatch 2024-06-06T08:26:02.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:02 smithi086 ceph-mon[29114]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f63fff7-ffe7-457f-a2c0-a5799c83ba32"}]: dispatch 2024-06-06T08:26:02.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:02 smithi086 ceph-mon[29114]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f63fff7-ffe7-457f-a2c0-a5799c83ba32"}]': finished 2024-06-06T08:26:02.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:02 smithi086 ceph-mon[29114]: osdmap e31: 5 total, 4 up, 5 in 2024-06-06T08:26:02.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:02 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:03.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:03 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/3633222537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:26:03.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:03 smithi086 ceph-mon[29114]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-06-06T08:26:03.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:03 smithi154 ceph-mon[35428]: from='client.? 172.21.15.154:0/3633222537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:26:03.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:03 smithi154 ceph-mon[35428]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-06-06T08:26:05.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:05 smithi154 ceph-mon[35428]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-06-06T08:26:06.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:05 smithi086 ceph-mon[29114]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-06-06T08:26:07.872 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:07 smithi154 ceph-mon[35428]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-06T08:26:08.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:07 smithi086 ceph-mon[29114]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-06T08:26:09.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:09 smithi154 ceph-mon[35428]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-06T08:26:09.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-06T08:26:09.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:09.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:09 smithi154 ceph-mon[35428]: Deploying daemon osd.4 on smithi154 2024-06-06T08:26:09.918 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:26:10.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:09 smithi086 ceph-mon[29114]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-06T08:26:10.139 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-06T08:26:10.139 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:10.139 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:09 smithi086 ceph-mon[29114]: Deploying daemon osd.4 on smithi154 2024-06-06T08:26:10.139 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:26:11.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:11 smithi154 ceph-mon[35428]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-06T08:26:12.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:11 smithi086 ceph-mon[29114]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-06T08:26:14.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:13 smithi086 ceph-mon[29114]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:14.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:13 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:26:14.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:13 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:14.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:13 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:14.145 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:13 smithi154 ceph-mon[35428]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:14.145 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:13 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:26:14.145 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:13 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:14.145 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:13 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:16.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:15 smithi086 ceph-mon[29114]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:16.157 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:15 smithi154 ceph-mon[35428]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:17.716 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.717+0000 7fe0417fa640 1 -- 172.21.15.154:0/2745433594 <== mgr.14212 v2:172.21.15.86:6800/3471926911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe04410eec0 con 0x7fe034065430 2024-06-06T08:26:17.717 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 4 on host 'smithi154' 2024-06-06T08:26:17.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe034065430 msgr2=0x7fe0340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:17.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe034065430 0x7fe0340678e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe038009f80 tx=0x7fe038009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:17.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fd560 msgr2=0x7fe04410d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:17.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fd560 0x7fe04410d460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe03000b790 tx=0x7fe0300091d0 comp rx=0 tx=0).stop 2024-06-06T08:26:17.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 shutdown_connections 2024-06-06T08:26:17.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0440fd560 0x7fe04410d460 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:17.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe034065430 0x7fe0340678e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:17.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 --2- 172.21.15.154:0/2745433594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0440fc290 0x7fe04410cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:17.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 >> 172.21.15.154:0/2745433594 conn(0x7fe0440f7a70 msgr2=0x7fe0440f90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:17.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 shutdown_connections 2024-06-06T08:26:17.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:17.719+0000 7fe04af1b640 1 -- 172.21.15.154:0/2745433594 wait complete. 2024-06-06T08:26:18.324 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:18.324 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.324 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.324 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:18.324 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:26:18.325 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.325 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='osd.4 [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T08:26:18.325 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T08:26:18.325 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:26:18.325 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.325 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:17 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:18.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:18.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:26:18.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='osd.4 [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T08:26:18.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T08:26:18.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:26:18.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:17 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:18.351 DEBUG:teuthology.orchestra.run.smithi154:osd.4> sudo journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.4.service 2024-06-06T08:26:18.353 INFO:tasks.cephadm:Deploying osd.5 on smithi154 with /dev/vg_nvme/lv_2... 2024-06-06T08:26:18.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-06T08:26:19.330 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:19 smithi154 ceph-mon[35428]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-06T08:26:19.331 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:19 smithi154 ceph-mon[35428]: osdmap e32: 5 total, 4 up, 5 in 2024-06-06T08:26:19.331 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:19 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:19.331 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:19 smithi154 ceph-mon[35428]: from='osd.4 [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:26:19.331 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:19 smithi154 ceph-mon[35428]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:26:19.331 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:26:19 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:26:19.014+0000 7fb89d93f640 -1 osd.4 0 waiting for initial osdmap 2024-06-06T08:26:19.331 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:26:19 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:26:19.021+0000 7fb898f77640 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T08:26:19.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:19 smithi086 ceph-mon[29114]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-06T08:26:19.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:19 smithi086 ceph-mon[29114]: osdmap e32: 5 total, 4 up, 5 in 2024-06-06T08:26:19.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:19 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:19.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:19 smithi086 ceph-mon[29114]: from='osd.4 [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:26:19.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:19 smithi086 ceph-mon[29114]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:26:19.379 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi154/config 2024-06-06T08:26:20.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:20 smithi154 ceph-mon[35428]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:20.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:20 smithi154 ceph-mon[35428]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-06-06T08:26:20.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:20 smithi154 ceph-mon[35428]: osdmap e33: 5 total, 4 up, 5 in 2024-06-06T08:26:20.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:20 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:20.168 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:20 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:20.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:20 smithi086 ceph-mon[29114]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:20.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:20 smithi086 ceph-mon[29114]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-06-06T08:26:20.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:20 smithi086 ceph-mon[29114]: osdmap e33: 5 total, 4 up, 5 in 2024-06-06T08:26:20.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:20 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:20.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:20 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:21.111 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:21 smithi154 ceph-mon[35428]: purged_snaps scrub starts 2024-06-06T08:26:21.112 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:21 smithi154 ceph-mon[35428]: purged_snaps scrub ok 2024-06-06T08:26:21.112 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:21 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:21.112 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:21 smithi154 ceph-mon[35428]: osd.4 [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] boot 2024-06-06T08:26:21.112 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:21 smithi154 ceph-mon[35428]: osdmap e34: 5 total, 5 up, 5 in 2024-06-06T08:26:21.112 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:21 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:21.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:21 smithi086 ceph-mon[29114]: purged_snaps scrub starts 2024-06-06T08:26:21.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:21 smithi086 ceph-mon[29114]: purged_snaps scrub ok 2024-06-06T08:26:21.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:21 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:21.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:21 smithi086 ceph-mon[29114]: osd.4 [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] boot 2024-06-06T08:26:21.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:21 smithi086 ceph-mon[29114]: osdmap e34: 5 total, 5 up, 5 in 2024-06-06T08:26:21.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:21 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T08:26:21.427 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-06T08:26:21.508 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-06T08:26:21.523 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-06-06T08:26:21.523 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-06-06T08:26:21.523 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124092 s, 845 MB/s 2024-06-06T08:26:21.523 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-06-06T08:26:22.135 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:22 smithi154 ceph-mon[35428]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:22.135 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:22 smithi154 ceph-mon[35428]: osdmap e35: 5 total, 5 up, 5 in 2024-06-06T08:26:22.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:22 smithi086 ceph-mon[29114]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T08:26:22.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:22 smithi086 ceph-mon[29114]: osdmap e35: 5 total, 5 up, 5 in 2024-06-06T08:26:22.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_2 2024-06-06T08:26:22.826 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi154/config 2024-06-06T08:26:23.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: Detected new or changed devices on smithi154 2024-06-06T08:26:23.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:23.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:23.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:23.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:23.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: Adjusting osd_memory_target on smithi154 to 7968M 2024-06-06T08:26:23.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:23.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:23.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:26:23.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:23.668 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:23 smithi154 ceph-mon[35428]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: Detected new or changed devices on smithi154 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: Adjusting osd_memory_target on smithi154 to 7968M 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:23.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:23 smithi086 ceph-mon[29114]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:24.243 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.241+0000 7fa321237640 1 -- 172.21.15.154:0/1121904489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fc480 msgr2=0x7fa31c0fc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:24.243 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.241+0000 7fa321237640 1 --2- 172.21.15.154:0/1121904489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fc480 0x7fa31c0fc8f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa3040048d0 tx=0x7fa304009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:24.243 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.241+0000 7fa321237640 1 -- 172.21.15.154:0/1121904489 shutdown_connections 2024-06-06T08:26:24.243 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.241+0000 7fa321237640 1 --2- 172.21.15.154:0/1121904489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fc480 0x7fa31c0fc8f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:24.243 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.241+0000 7fa321237640 1 --2- 172.21.15.154:0/1121904489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c0fb1b0 0x7fa31c0fb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.241+0000 7fa321237640 1 -- 172.21.15.154:0/1121904489 >> 172.21.15.154:0/1121904489 conn(0x7fa31c0f6970 msgr2=0x7fa31c0f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa321237640 1 -- 172.21.15.154:0/1121904489 shutdown_connections 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa321237640 1 -- 172.21.15.154:0/1121904489 wait complete. 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa321237640 1 Processor -- start 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa321237640 1 -- start start 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa321237640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fb1b0 0x7fa31c10cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa321237640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c0fc480 0x7fa31c10d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa321237640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c10daa0 con 0x7fa31c0fc480 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa321237640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c10e2c0 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa31ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fb1b0 0x7fa31c10cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:24.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa31ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fb1b0 0x7fa31c10cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:58872/0 (socket says 172.21.15.154:58872) 2024-06-06T08:26:24.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa31ad9d640 1 -- 172.21.15.154:0/128913278 learned_addr learned my addr 172.21.15.154:0/128913278 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:26:24.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.242+0000 7fa313fff640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c0fc480 0x7fa31c10d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:24.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.243+0000 7fa31ad9d640 1 -- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c0fc480 msgr2=0x7fa31c10d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:24.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.243+0000 7fa31ad9d640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c0fc480 0x7fa31c10d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:24.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.243+0000 7fa31ad9d640 1 -- 172.21.15.154:0/128913278 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa304005040 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.243+0000 7fa31ad9d640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fb1b0 0x7fa31c10cf10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa30c002a50 tx=0x7fa30c004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:24.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.243+0000 7fa3137fe640 1 -- 172.21.15.154:0/128913278 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa30c007c30 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.243+0000 7fa3137fe640 1 -- 172.21.15.154:0/128913278 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa30c00f040 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.243+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c10e550 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.243+0000 7fa3137fe640 1 -- 172.21.15.154:0/128913278 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa30c00dad0 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.243+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c10eac0 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.244+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31c104530 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.247 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.244+0000 7fa3137fe640 1 -- 172.21.15.154:0/128913278 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa30c00dd40 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.247 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.245+0000 7fa3137fe640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa308065430 0x7fa3080678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:24.247 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.245+0000 7fa3137fe640 1 -- 172.21.15.154:0/128913278 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7fa30c081ae0 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.247 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.245+0000 7fa313fff640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa308065430 0x7fa3080678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:24.247 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.246+0000 7fa313fff640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa308065430 0x7fa3080678e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa3040043d0 tx=0x7fa304035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:24.248 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.249+0000 7fa3137fe640 1 -- 172.21.15.154:0/128913278 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa30c04cdf0 con 0x7fa31c0fb1b0 2024-06-06T08:26:24.420 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:24.420+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 --> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa31c103ed0 con 0x7fa308065430 2024-06-06T08:26:24.828 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:24 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:26:24.829 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:24 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:26:24.829 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:24 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:24.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T08:26:24.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T08:26:24.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:25.701 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:25 smithi154 ceph-mon[35428]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:26:25.701 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:25 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:26:25.701 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:25 smithi154 ceph-mon[35428]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:25.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:25 smithi086 ceph-mon[29114]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T08:26:25.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:25 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:26:25.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:25 smithi086 ceph-mon[29114]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:27.028 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:26 smithi154 ceph-mon[35428]: from='client.? 172.21.15.154:0/1861271350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194e8288-0427-44ad-8b0d-dc8c67aee3ce"}]: dispatch 2024-06-06T08:26:27.029 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:26 smithi154 ceph-mon[35428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194e8288-0427-44ad-8b0d-dc8c67aee3ce"}]: dispatch 2024-06-06T08:26:27.029 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:26 smithi154 ceph-mon[35428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "194e8288-0427-44ad-8b0d-dc8c67aee3ce"}]': finished 2024-06-06T08:26:27.029 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:26 smithi154 ceph-mon[35428]: osdmap e36: 6 total, 5 up, 6 in 2024-06-06T08:26:27.029 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:26 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:26 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/1861271350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194e8288-0427-44ad-8b0d-dc8c67aee3ce"}]: dispatch 2024-06-06T08:26:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:26 smithi086 ceph-mon[29114]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194e8288-0427-44ad-8b0d-dc8c67aee3ce"}]: dispatch 2024-06-06T08:26:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:26 smithi086 ceph-mon[29114]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "194e8288-0427-44ad-8b0d-dc8c67aee3ce"}]': finished 2024-06-06T08:26:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:26 smithi086 ceph-mon[29114]: osdmap e36: 6 total, 5 up, 6 in 2024-06-06T08:26:27.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:26 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:28.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:27 smithi086 ceph-mon[29114]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:28.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:27 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/2848321884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:26:28.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:27 smithi154 ceph-mon[35428]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:28.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:27 smithi154 ceph-mon[35428]: from='client.? 172.21.15.154:0/2848321884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T08:26:30.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:29 smithi086 ceph-mon[29114]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:30.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:29 smithi154 ceph-mon[35428]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:31.898 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:31 smithi154 ceph-mon[35428]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:32.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:31 smithi086 ceph-mon[29114]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:34.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:33 smithi086 ceph-mon[29114]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:34.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:33 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-06T08:26:34.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:33 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:34.133 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:33 smithi154 ceph-mon[35428]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:34.133 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:33 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-06T08:26:34.133 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:33 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:35.019 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:34 smithi154 ceph-mon[35428]: Deploying daemon osd.5 on smithi154 2024-06-06T08:26:35.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:34 smithi086 ceph-mon[29114]: Deploying daemon osd.5 on smithi154 2024-06-06T08:26:35.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:35 smithi154 ceph-mon[35428]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:36.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:35 smithi086 ceph-mon[29114]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:38.059 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:37 smithi154 ceph-mon[35428]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:38.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:37 smithi086 ceph-mon[29114]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:39.065 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:26:39.065 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:39.065 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:38 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:39.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:26:39.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:39.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:38 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:40.065 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:39 smithi154 ceph-mon[35428]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:40.066 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:39 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:26:40.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:39 smithi086 ceph-mon[29114]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:40.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:39 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:26:42.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:41 smithi086 ceph-mon[29114]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:42.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:41 smithi154 ceph-mon[35428]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:42.450 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 5 on host 'smithi154' 2024-06-06T08:26:42.451 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.451+0000 7fa3137fe640 1 -- 172.21.15.154:0/128913278 <== mgr.14212 v2:172.21.15.86:6800/3471926911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa304039040 con 0x7fa308065430 2024-06-06T08:26:42.452 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa308065430 msgr2=0x7fa3080678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:42.452 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa308065430 0x7fa3080678e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa3040043d0 tx=0x7fa304035040 comp rx=0 tx=0).stop 2024-06-06T08:26:42.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fb1b0 msgr2=0x7fa31c10cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:42.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fb1b0 0x7fa31c10cf10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa30c002a50 tx=0x7fa30c004600 comp rx=0 tx=0).stop 2024-06-06T08:26:42.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 shutdown_connections 2024-06-06T08:26:42.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa308065430 0x7fa3080678e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:42.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa31c0fc480 0x7fa31c10d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:42.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 --2- 172.21.15.154:0/128913278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31c0fb1b0 0x7fa31c10cf10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:42.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 >> 172.21.15.154:0/128913278 conn(0x7fa31c0f6970 msgr2=0x7fa31c065dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:42.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.453+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 shutdown_connections 2024-06-06T08:26:42.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:42.454+0000 7fa321237640 1 -- 172.21.15.154:0/128913278 wait complete. 2024-06-06T08:26:43.070 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:42 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:43.070 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:42 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:43.070 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:42 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:43.070 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:42 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:43.070 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:42 smithi154 ceph-mon[35428]: from='osd.5 [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T08:26:43.070 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:42 smithi154 ceph-mon[35428]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T08:26:43.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:42 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:43.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:42 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:43.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:42 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:43.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:42 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:43.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:42 smithi086 ceph-mon[29114]: from='osd.5 [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T08:26:43.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:42 smithi086 ceph-mon[29114]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T08:26:43.094 DEBUG:teuthology.orchestra.run.smithi154:osd.5> sudo journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.5.service 2024-06-06T08:26:43.096 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-06-06T08:26:43.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-06T08:26:43.367 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:26:43.906 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:43 smithi154 ceph-mon[35428]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:43.906 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:43 smithi154 ceph-mon[35428]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-06T08:26:43.907 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:43 smithi154 ceph-mon[35428]: osdmap e37: 6 total, 5 up, 6 in 2024-06-06T08:26:43.907 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:43 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:43.907 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:43 smithi154 ceph-mon[35428]: from='osd.5 [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:26:43.907 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:43 smithi154 ceph-mon[35428]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:26:44.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:43 smithi086 ceph-mon[29114]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:44.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:43 smithi086 ceph-mon[29114]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-06T08:26:44.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:43 smithi086 ceph-mon[29114]: osdmap e37: 6 total, 5 up, 6 in 2024-06-06T08:26:44.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:43 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:44.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:43 smithi086 ceph-mon[29114]: from='osd.5 [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:26:44.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:43 smithi086 ceph-mon[29114]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-06-06T08:26:44.698 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:26:44 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:26:44.460+0000 7fad4f9e9640 -1 osd.5 0 waiting for initial osdmap 2024-06-06T08:26:44.699 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:26:44 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:26:44.468+0000 7fad4a820640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T08:26:45.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 -- 172.21.15.86:0/1789228185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fd550 msgr2=0x7f77880fd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:45.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 --2- 172.21.15.86:0/1789228185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fd550 0x7f77880fd9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f77840048d0 tx=0x7f7784009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:45.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 -- 172.21.15.86:0/1789228185 shutdown_connections 2024-06-06T08:26:45.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 --2- 172.21.15.86:0/1789228185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fd550 0x7f77880fd9c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 --2- 172.21.15.86:0/1789228185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77880fc280 0x7f77880fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 -- 172.21.15.86:0/1789228185 >> 172.21.15.86:0/1789228185 conn(0x7f77880f7a40 msgr2=0x7f77880f9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 -- 172.21.15.86:0/1789228185 shutdown_connections 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 -- 172.21.15.86:0/1789228185 wait complete. 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 Processor -- start 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.029+0000 7f778faf9640 1 -- start start 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778faf9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fc280 0x7f778810cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778faf9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77880fd550 0x7f778810d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778faf9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778810e2c0 con 0x7f77880fc280 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778faf9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778810e430 con 0x7f77880fd550 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778d895640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fc280 0x7f778810cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778d895640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fc280 0x7f778810cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56252/0 (socket says 172.21.15.86:56252) 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778d895640 1 -- 172.21.15.86:0/3642451971 learned_addr learned my addr 172.21.15.86:0/3642451971 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778d094640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77880fd550 0x7f778810d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778d895640 1 -- 172.21.15.86:0/3642451971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77880fd550 msgr2=0x7f778810d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778d895640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77880fd550 0x7f778810d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778d895640 1 -- 172.21.15.86:0/3642451971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7784005040 con 0x7f77880fc280 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778d094640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77880fd550 0x7f778810d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778d895640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fc280 0x7f778810cf10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f777c00b790 tx=0x7f777c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f777a7fc640 1 -- 172.21.15.86:0/3642451971 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f777c00c700 con 0x7f77880fc280 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778810e710 con 0x7f77880fc280 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f777a7fc640 1 -- 172.21.15.86:0/3642451971 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f777c00cd30 con 0x7f77880fc280 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f777a7fc640 1 -- 172.21.15.86:0/3642451971 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f777c013370 con 0x7f77880fc280 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.030+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778810ec30 con 0x7f77880fc280 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.031+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7788105660 con 0x7f77880fc280 2024-06-06T08:26:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.032+0000 7f777a7fc640 1 -- 172.21.15.86:0/3642451971 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f777c012070 con 0x7f77880fc280 2024-06-06T08:26:45.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.032+0000 7f777a7fc640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7774065430 0x7f77740678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:45.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.032+0000 7f778d094640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7774065430 0x7f77740678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:45.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.032+0000 7f777a7fc640 1 -- 172.21.15.86:0/3642451971 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f777c081fc0 con 0x7f77880fc280 2024-06-06T08:26:45.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.033+0000 7f778d094640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7774065430 0x7f77740678e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f77840047f0 tx=0x7f7784009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:45.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.037+0000 7f777a7fc640 1 -- 172.21.15.86:0/3642451971 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f777c050580 con 0x7f77880fc280 2024-06-06T08:26:45.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.201+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f77880fb6e0 con 0x7f77880fc280 2024-06-06T08:26:45.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.201+0000 7f777a7fc640 1 -- 172.21.15.86:0/3642451971 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f777c053ba0 con 0x7f77880fc280 2024-06-06T08:26:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:26:45.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7774065430 msgr2=0x7f77740678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7774065430 0x7f77740678e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f77840047f0 tx=0x7f7784009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fc280 msgr2=0x7f778810cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fc280 0x7f778810cf10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f777c00b790 tx=0x7f777c0091d0 comp rx=0 tx=0).stop 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 shutdown_connections 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77880fd550 0x7f778810d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7774065430 0x7f77740678e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 --2- 172.21.15.86:0/3642451971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77880fc280 0x7f778810cf10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 >> 172.21.15.86:0/3642451971 conn(0x7f77880f7a40 msgr2=0x7f77880f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 shutdown_connections 2024-06-06T08:26:45.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:45.203+0000 7f778faf9640 1 -- 172.21.15.86:0/3642451971 wait complete. 2024-06-06T08:26:45.743 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:45 smithi154 ceph-mon[35428]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-06-06T08:26:45.743 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:45 smithi154 ceph-mon[35428]: osdmap e38: 6 total, 5 up, 6 in 2024-06-06T08:26:45.743 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:45 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:45.743 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:45 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:45.744 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:45 smithi154 ceph-mon[35428]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:45.744 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:45 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3642451971' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T08:26:45.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:45 smithi086 ceph-mon[29114]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-06-06T08:26:45.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:45 smithi086 ceph-mon[29114]: osdmap e38: 6 total, 5 up, 6 in 2024-06-06T08:26:45.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:45 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:45.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:45 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:45.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:45 smithi086 ceph-mon[29114]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T08:26:45.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:45 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3642451971' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T08:26:45.905 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1717662380,"num_in_osds":6,"osd_in_since":1717662386,"num_remapped_pgs":0} 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: purged_snaps scrub starts 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: purged_snaps scrub ok 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: osd.5 [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] boot 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: osdmap e39: 6 total, 6 up, 6 in 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: Detected new or changed devices on smithi154 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:46.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:46.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: Adjusting osd_memory_target on smithi154 to 5312M 2024-06-06T08:26:46.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:46.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:46.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:26:46.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:46.836 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:46 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:26:46.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-06T08:26:46.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: purged_snaps scrub starts 2024-06-06T08:26:46.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: purged_snaps scrub ok 2024-06-06T08:26:46.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: osd.5 [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] boot 2024-06-06T08:26:46.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: osdmap e39: 6 total, 6 up, 6 in 2024-06-06T08:26:46.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T08:26:46.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: Detected new or changed devices on smithi154 2024-06-06T08:26:46.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:46.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:46.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:46.918 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:46.918 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-06T08:26:46.918 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: Adjusting osd_memory_target on smithi154 to 5312M 2024-06-06T08:26:46.918 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:46.918 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:46.918 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:26:46.918 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:46.918 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:46 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:26:47.156 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:26:47.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:47 smithi086 ceph-mon[29114]: osdmap e40: 6 total, 6 up, 6 in 2024-06-06T08:26:47.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:47 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:47.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:47 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:26:47.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:47 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:47.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:47 smithi086 ceph-mon[29114]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:47.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:47 smithi154 ceph-mon[35428]: osdmap e40: 6 total, 6 up, 6 in 2024-06-06T08:26:47.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:47 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:26:47.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:47 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:26:47.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:47 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:26:47.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:47 smithi154 ceph-mon[35428]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 -- 172.21.15.86:0/110386838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fbc10 msgr2=0x7f131c0fc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 --2- 172.21.15.86:0/110386838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fbc10 0x7f131c0fc080 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f13040048d0 tx=0x7f1304009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 -- 172.21.15.86:0/110386838 shutdown_connections 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 --2- 172.21.15.86:0/110386838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fbc10 0x7f131c0fc080 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 --2- 172.21.15.86:0/110386838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c0fa940 0x7f131c0fad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 -- 172.21.15.86:0/110386838 >> 172.21.15.86:0/110386838 conn(0x7f131c0f6100 msgr2=0x7f131c0f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 -- 172.21.15.86:0/110386838 shutdown_connections 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 -- 172.21.15.86:0/110386838 wait complete. 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 Processor -- start 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 -- start start 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fa940 0x7f131c10cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c0fbc10 0x7f131c10d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c10e310 con 0x7f131c0fa940 2024-06-06T08:26:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.802+0000 7f1320f98640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c10e480 con 0x7f131c0fbc10 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f131a59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fa940 0x7f131c10cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f131a59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fa940 0x7f131c10cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58586/0 (socket says 172.21.15.86:58586) 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f131a59c640 1 -- 172.21.15.86:0/1141003682 learned_addr learned my addr 172.21.15.86:0/1141003682 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f1313fff640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c0fbc10 0x7f131c10d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f131a59c640 1 -- 172.21.15.86:0/1141003682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c0fbc10 msgr2=0x7f131c10d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f131a59c640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c0fbc10 0x7f131c10d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f131a59c640 1 -- 172.21.15.86:0/1141003682 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304005040 con 0x7f131c0fa940 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f1313fff640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c0fbc10 0x7f131c10d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f131a59c640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fa940 0x7f131c10cf60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f130c00b790 tx=0x7f130c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f1312ffd640 1 -- 172.21.15.86:0/1141003682 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f130c00c700 con 0x7f131c0fa940 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f1312ffd640 1 -- 172.21.15.86:0/1141003682 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f130c00cd30 con 0x7f131c0fa940 2024-06-06T08:26:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f1312ffd640 1 -- 172.21.15.86:0/1141003682 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f130c013370 con 0x7f131c0fa940 2024-06-06T08:26:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c10e760 con 0x7f131c0fa940 2024-06-06T08:26:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.803+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c10ec30 con 0x7f131c0fa940 2024-06-06T08:26:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.804+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c103d20 con 0x7f131c0fa940 2024-06-06T08:26:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.804+0000 7f1312ffd640 1 -- 172.21.15.86:0/1141003682 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f130c012070 con 0x7f131c0fa940 2024-06-06T08:26:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.805+0000 7f1312ffd640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1308065430 0x7f13080678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.805+0000 7f1312ffd640 1 -- 172.21.15.86:0/1141003682 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f130c0820d0 con 0x7f131c0fa940 2024-06-06T08:26:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.805+0000 7f1313fff640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1308065430 0x7f13080678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.805+0000 7f1313fff640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1308065430 0x7f13080678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f13040047f0 tx=0x7f1304009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.809+0000 7f1312ffd640 1 -- 172.21.15.86:0/1141003682 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f130c050580 con 0x7f131c0fa940 2024-06-06T08:26:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.974+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f131c0f9da0 con 0x7f131c0fa940 2024-06-06T08:26:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.974+0000 7f1312ffd640 1 -- 172.21.15.86:0/1141003682 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f130c053ba0 con 0x7f131c0fa940 2024-06-06T08:26:48.974 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:26:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1308065430 msgr2=0x7f13080678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1308065430 0x7f13080678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f13040047f0 tx=0x7f1304009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fa940 msgr2=0x7f131c10cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fa940 0x7f131c10cf60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f130c00b790 tx=0x7f130c0091d0 comp rx=0 tx=0).stop 2024-06-06T08:26:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 shutdown_connections 2024-06-06T08:26:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c0fbc10 0x7f131c10d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1308065430 0x7f13080678e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 --2- 172.21.15.86:0/1141003682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f131c0fa940 0x7f131c10cf60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 >> 172.21.15.86:0/1141003682 conn(0x7f131c0f6100 msgr2=0x7f131c0f7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 shutdown_connections 2024-06-06T08:26:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:48.977+0000 7f1320f98640 1 -- 172.21.15.86:0/1141003682 wait complete. 2024-06-06T08:26:49.661 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1717662405,"num_in_osds":6,"osd_in_since":1717662386,"num_remapped_pgs":0} 2024-06-06T08:26:49.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-06T08:26:49.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:49 smithi086 ceph-mon[29114]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:49.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:49 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1141003682' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T08:26:49.914 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:26:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:49 smithi154 ceph-mon[35428]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:49 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1141003682' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T08:26:51.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.568+0000 7fe600075640 1 -- 172.21.15.86:0/72393038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 msgr2=0x7fe5f80fbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:51.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.568+0000 7fe600075640 1 --2- 172.21.15.86:0/72393038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 0x7fe5f80fbb60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80048d0 tx=0x7fe5e8009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:51.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.568+0000 7fe600075640 1 -- 172.21.15.86:0/72393038 shutdown_connections 2024-06-06T08:26:51.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.568+0000 7fe600075640 1 --2- 172.21.15.86:0/72393038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 0x7fe5f80fbb60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:51.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.568+0000 7fe600075640 1 --2- 172.21.15.86:0/72393038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f81012c0 0x7fe5f8101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:51.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.568+0000 7fe600075640 1 -- 172.21.15.86:0/72393038 >> 172.21.15.86:0/72393038 conn(0x7fe5f8070ab0 msgr2=0x7fe5f8070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:51.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.568+0000 7fe600075640 1 -- 172.21.15.86:0/72393038 shutdown_connections 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.568+0000 7fe600075640 1 -- 172.21.15.86:0/72393038 wait complete. 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe600075640 1 Processor -- start 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe600075640 1 -- start start 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe600075640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f81012c0 0x7fe5f81020d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe600075640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 0x7fe5f8102610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe600075640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f8102be0 con 0x7fe5f8101bd0 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe600075640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f8102d50 con 0x7fe5f81012c0 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe5fd610640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 0x7fe5f8102610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe5fd610640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 0x7fe5f8102610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58604/0 (socket says 172.21.15.86:58604) 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe5fd610640 1 -- 172.21.15.86:0/4766104 learned_addr learned my addr 172.21.15.86:0/4766104 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe5fd610640 1 -- 172.21.15.86:0/4766104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f81012c0 msgr2=0x7fe5f81020d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe5fd610640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f81012c0 0x7fe5f81020d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe5fd610640 1 -- 172.21.15.86:0/4766104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8005040 con 0x7fe5f8101bd0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.569+0000 7fe5fd610640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 0x7fe5f8102610 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8032430 tx=0x7fe5e8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.570+0000 7fe5ee7fc640 1 -- 172.21.15.86:0/4766104 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5e8038070 con 0x7fe5f8101bd0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.570+0000 7fe5ee7fc640 1 -- 172.21.15.86:0/4766104 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5e8032d60 con 0x7fe5f8101bd0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.570+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f8119420 con 0x7fe5f8101bd0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.570+0000 7fe5ee7fc640 1 -- 172.21.15.86:0/4766104 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5e803de10 con 0x7fe5f8101bd0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.570+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f81197a0 con 0x7fe5f8101bd0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.571+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f80fcb70 con 0x7fe5f8101bd0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.571+0000 7fe5ee7fc640 1 -- 172.21.15.86:0/4766104 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe5e8030020 con 0x7fe5f8101bd0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.571+0000 7fe5ee7fc640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe5e4065430 0x7fe5e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.572+0000 7fe5ee7fc640 1 -- 172.21.15.86:0/4766104 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe5e80a2fe0 con 0x7fe5f8101bd0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.572+0000 7fe5fde11640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe5e4065430 0x7fe5e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.572+0000 7fe5fde11640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe5e4065430 0x7fe5e40678e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40097e0 tx=0x7fe5f4006d20 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.576+0000 7fe5ee7fc640 1 -- 172.21.15.86:0/4766104 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe5e8071490 con 0x7fe5f8101bd0 2024-06-06T08:26:51.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.744+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5f80749d0 con 0x7fe5f8101bd0 2024-06-06T08:26:51.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.744+0000 7fe5ee7fc640 1 -- 172.21.15.86:0/4766104 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11081 (secure 0 0 0) 0x7fe5e8074ab0 con 0x7fe5f8101bd0 2024-06-06T08:26:51.744 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:26:51.744 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":40,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","created":"2024-06-06T08:21:26.882664+0000","modified":"2024-06-06T08:26:46.462722+0000","last_up_change":"2024-06-06T08:26:45.455244+0000","last_in_change":"2024-06-06T08:26:26.696925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-06T08:25:32.951773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"50d6c551-bd2c-4826-a0e9-faf49f3759d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6803","nonce":771268483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6805","nonce":771268483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6809","nonce":771268483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6807","nonce":771268483}]},"public_addr":"172.21.15.86:6803/771268483","cluster_addr":"172.21.15.86:6805/771268483","heartbeat_back_addr":"172.21.15.86:6809/771268483","heartbeat_front_addr":"172.21.15.86:6807/771268483","state":["exists","up"]},{"osd":1,"uuid":"6e6e7221-c7a2-48e9-b584-83b541395665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6811","nonce":1628660505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6813","nonce":1628660505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6817","nonce":1628660505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6815","nonce":1628660505}]},"public_addr":"172.21.15.86:6811/1628660505","cluster_addr":"172.21.15.86:6813/1628660505","heartbeat_back_addr":"172.21.15.86:6817/1628660505","heartbeat_front_addr":"172.21.15.86:6815/1628660505","state":["exists","up"]},{"osd":2,"uuid":"d2ad1948-0ddb-4821-b150-f723c3877245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6819","nonce":3769521712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6821","nonce":3769521712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6825","nonce":3769521712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6823","nonce":3769521712}]},"public_addr":"172.21.15.86:6819/3769521712","cluster_addr":"172.21.15.86:6821/3769521712","heartbeat_back_addr":"172.21.15.86:6825/3769521712","heartbeat_front_addr":"172.21.15.86:6823/3769521712","state":["exists","up"]},{"osd":3,"uuid":"197f72f4-3b9e-4f36-a8fe-854cc214c6ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6801","nonce":1068971095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6803","nonce":1068971095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6807","nonce":1068971095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6805","nonce":1068971095}]},"public_addr":"172.21.15.154:6801/1068971095","cluster_addr":"172.21.15.154:6803/1068971095","heartbeat_back_addr":"172.21.15.154:6807/1068971095","heartbeat_front_addr":"172.21.15.154:6805/1068971095","state":["exists","up"]},{"osd":4,"uuid":"3f63fff7-ffe7-457f-a2c0-a5799c83ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6809","nonce":3195305408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6811","nonce":3195305408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6815","nonce":3195305408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6813","nonce":3195305408}]},"public_addr":"172.21.15.154:6809/3195305408","cluster_addr":"172.21.15.154:6811/3195305408","heartbeat_back_addr":"172.21.15.154:6815/3195305408","heartbeat_front_addr":"172.21.15.154:6813/3195305408","state":["exists","up"]},{"osd":5,"uuid":"194e8288-0427-44ad-8b0d-dc8c67aee3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6817","nonce":2218758456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6819","nonce":2218758456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6823","nonce":2218758456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6821","nonce":2218758456}]},"public_addr":"172.21.15.154:6817/2218758456","cluster_addr":"172.21.15.154:6819/2218758456","heartbeat_back_addr":"172.21.15.154:6823/2218758456","heartbeat_front_addr":"172.21.15.154:6821/2218758456","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:24:36.527260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:25:03.405907+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:25:30.733020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:25:54.062475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:26:18.638828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:26:43.487776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2470791934":"2024-06-07T08:23:24.624760+0000","172.21.15.86:6800/2470791934":"2024-06-07T08:23:24.624760+0000","172.21.15.86:6801/3222204828":"2024-06-07T08:21:56.545284+0000","172.21.15.86:6800/3222204828":"2024-06-07T08:21:56.545284+0000","172.21.15.86:0/3370799480":"2024-06-07T08:21:56.545284+0000","172.21.15.86:6801/933334455":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/3574006712":"2024-06-07T08:21:56.545284+0000","172.21.15.86:0/2909599705":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/2894717502":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/2181614567":"2024-06-07T08:21:56.545284+0000","172.21.15.86:6800/933334455":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/808628117":"2024-06-07T08:23:24.624760+0000","172.21.15.86:0/3541586639":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/2450050335":"2024-06-07T08:23:24.624760+0000","172.21.15.86:0/753216971":"2024-06-07T08:23:24.624760+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-06T08:26:51.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe5e4065430 msgr2=0x7fe5e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe5e4065430 0x7fe5e40678e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40097e0 tx=0x7fe5f4006d20 comp rx=0 tx=0).stop 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 msgr2=0x7fe5f8102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 0x7fe5f8102610 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8032430 tx=0x7fe5e8009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 shutdown_connections 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe5e4065430 0x7fe5e40678e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f8101bd0 0x7fe5f8102610 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 --2- 172.21.15.86:0/4766104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f81012c0 0x7fe5f81020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 >> 172.21.15.86:0/4766104 conn(0x7fe5f8070ab0 msgr2=0x7fe5f8104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 shutdown_connections 2024-06-06T08:26:51.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:51.746+0000 7fe600075640 1 -- 172.21.15.86:0/4766104 wait complete. 2024-06-06T08:26:52.065 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:51 smithi086 ceph-mon[29114]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:52.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:51 smithi154 ceph-mon[35428]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:52.491 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-06T08:25:32.951773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-06T08:26:52.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-06T08:26:52.742 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:26:52.993 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:52 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/4766104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T08:26:53.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:52 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/4766104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T08:26:54.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:53 smithi086 ceph-mon[29114]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:54.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:53 smithi154 ceph-mon[35428]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:54.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.405+0000 7ff4a644c640 1 -- 172.21.15.86:0/3981166998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a01022c0 msgr2=0x7ff4a0102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:54.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.405+0000 7ff4a644c640 1 --2- 172.21.15.86:0/3981166998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a01022c0 0x7ff4a0102690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff4900048d0 tx=0x7ff490009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.405+0000 7ff4a644c640 1 -- 172.21.15.86:0/3981166998 shutdown_connections 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.405+0000 7ff4a644c640 1 --2- 172.21.15.86:0/3981166998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a0102c60 0x7ff4a00fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.405+0000 7ff4a644c640 1 --2- 172.21.15.86:0/3981166998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a01022c0 0x7ff4a0102690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.405+0000 7ff4a644c640 1 -- 172.21.15.86:0/3981166998 >> 172.21.15.86:0/3981166998 conn(0x7ff4a00f7e50 msgr2=0x7ff4a00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.405+0000 7ff4a644c640 1 -- 172.21.15.86:0/3981166998 shutdown_connections 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.405+0000 7ff4a644c640 1 -- 172.21.15.86:0/3981166998 wait complete. 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff4a644c640 1 Processor -- start 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff4a644c640 1 -- start start 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff4a644c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a01022c0 0x7ff4a0113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff4a644c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a0102c60 0x7ff4a0113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff4a644c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0113e70 con 0x7ff4a0102c60 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff4a644c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a010d2b0 con 0x7ff4a01022c0 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff497fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a0102c60 0x7ff4a0113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:54.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff497fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a0102c60 0x7ff4a0113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58622/0 (socket says 172.21.15.86:58622) 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff497fff640 1 -- 172.21.15.86:0/458706624 learned_addr learned my addr 172.21.15.86:0/458706624 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff49ffff640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a01022c0 0x7ff4a0113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff497fff640 1 -- 172.21.15.86:0/458706624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a01022c0 msgr2=0x7ff4a0113330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff497fff640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a01022c0 0x7ff4a0113330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.406+0000 7ff497fff640 1 -- 172.21.15.86:0/458706624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff490005040 con 0x7ff4a0102c60 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.407+0000 7ff497fff640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a0102c60 0x7ff4a0113870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff48800c810 tx=0x7ff4880043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.407+0000 7ff49d7fa640 1 -- 172.21.15.86:0/458706624 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff48800a7a0 con 0x7ff4a0102c60 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.407+0000 7ff49d7fa640 1 -- 172.21.15.86:0/458706624 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff48800add0 con 0x7ff4a0102c60 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.407+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a010d590 con 0x7ff4a0102c60 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.407+0000 7ff49d7fa640 1 -- 172.21.15.86:0/458706624 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4880115c0 con 0x7ff4a0102c60 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.407+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a010db30 con 0x7ff4a0102c60 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.408+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a00fda30 con 0x7ff4a0102c60 2024-06-06T08:26:54.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.408+0000 7ff49d7fa640 1 -- 172.21.15.86:0/458706624 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff488012420 con 0x7ff4a0102c60 2024-06-06T08:26:54.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.409+0000 7ff49d7fa640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff48c065430 0x7ff48c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:54.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.409+0000 7ff49d7fa640 1 -- 172.21.15.86:0/458706624 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff488082970 con 0x7ff4a0102c60 2024-06-06T08:26:54.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.409+0000 7ff49ffff640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff48c065430 0x7ff48c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:54.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.409+0000 7ff49ffff640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff48c065430 0x7ff48c0678e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff490009f80 tx=0x7ff490035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:54.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.413+0000 7ff49d7fa640 1 -- 172.21.15.86:0/458706624 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff488050e20 con 0x7ff4a0102c60 2024-06-06T08:26:54.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.575+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff4a010e9a0 con 0x7ff4a0102c60 2024-06-06T08:26:54.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.575+0000 7ff49d7fa640 1 -- 172.21.15.86:0/458706624 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff488054440 con 0x7ff4a0102c60 2024-06-06T08:26:54.574 INFO:teuthology.orchestra.run.smithi086.stdout:pg_num: 1 2024-06-06T08:26:54.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff48c065430 msgr2=0x7ff48c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:54.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff48c065430 0x7ff48c0678e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff490009f80 tx=0x7ff490035040 comp rx=0 tx=0).stop 2024-06-06T08:26:54.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a0102c60 msgr2=0x7ff4a0113870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:54.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a0102c60 0x7ff4a0113870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff48800c810 tx=0x7ff4880043c0 comp rx=0 tx=0).stop 2024-06-06T08:26:54.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 shutdown_connections 2024-06-06T08:26:54.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff48c065430 0x7ff48c0678e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:54.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a0102c60 0x7ff4a0113870 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:54.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 --2- 172.21.15.86:0/458706624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a01022c0 0x7ff4a0113330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:54.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 >> 172.21.15.86:0/458706624 conn(0x7ff4a00f7e50 msgr2=0x7ff4a00f8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:54.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 shutdown_connections 2024-06-06T08:26:54.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:54.577+0000 7ff4a644c640 1 -- 172.21.15.86:0/458706624 wait complete. 2024-06-06T08:26:54.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:54 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/458706624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-06T08:26:54.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:26:55.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:54 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/458706624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-06T08:26:55.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:26:55.203 INFO:tasks.cephadm:Setting up client nodes... 2024-06-06T08:26:55.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-06T08:26:55.455 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:26:55.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:55 smithi086 ceph-mon[29114]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:56.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:55 smithi154 ceph-mon[35428]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.949+0000 7f92f293f640 1 -- 172.21.15.86:0/3924305026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 msgr2=0x7f92ec0fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.949+0000 7f92f293f640 1 --2- 172.21.15.86:0/3924305026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 0x7f92ec0fb580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f92d40048d0 tx=0x7f92d4009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.949+0000 7f92f293f640 1 -- 172.21.15.86:0/3924305026 shutdown_connections 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.949+0000 7f92f293f640 1 --2- 172.21.15.86:0/3924305026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 0x7f92ec0fb580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.949+0000 7f92f293f640 1 --2- 172.21.15.86:0/3924305026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92ec0fc800 0x7f92ec0fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.949+0000 7f92f293f640 1 -- 172.21.15.86:0/3924305026 >> 172.21.15.86:0/3924305026 conn(0x7f92ec0f6120 msgr2=0x7f92ec0f8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.949+0000 7f92f293f640 1 -- 172.21.15.86:0/3924305026 shutdown_connections 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.949+0000 7f92f293f640 1 -- 172.21.15.86:0/3924305026 wait complete. 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.949+0000 7f92f293f640 1 Processor -- start 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92f293f640 1 -- start start 2024-06-06T08:26:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92f293f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92ec0fc800 0x7f92ec10fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92f293f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 0x7f92ec1121b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92f293f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec0fb150 con 0x7f92ec0fd150 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92f293f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec0fb2c0 con 0x7f92ec0fc800 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92e3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 0x7f92ec1121b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92e3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 0x7f92ec1121b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58634/0 (socket says 172.21.15.86:58634) 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92e3fff640 1 -- 172.21.15.86:0/4260271982 learned_addr learned my addr 172.21.15.86:0/4260271982 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92ebfff640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92ec0fc800 0x7f92ec10fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92e3fff640 1 -- 172.21.15.86:0/4260271982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92ec0fc800 msgr2=0x7f92ec10fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92e3fff640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92ec0fc800 0x7f92ec10fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92e3fff640 1 -- 172.21.15.86:0/4260271982 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d4005040 con 0x7f92ec0fd150 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92e3fff640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 0x7f92ec1121b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009f80 tx=0x7f92d4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.950+0000 7f92e97fa640 1 -- 172.21.15.86:0/4260271982 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92d4038070 con 0x7f92ec0fd150 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.951+0000 7f92e97fa640 1 -- 172.21.15.86:0/4260271982 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92d4008ee0 con 0x7f92ec0fd150 2024-06-06T08:26:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.951+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ec1126f0 con 0x7f92ec0fd150 2024-06-06T08:26:56.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.951+0000 7f92e97fa640 1 -- 172.21.15.86:0/4260271982 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92d4033890 con 0x7f92ec0fd150 2024-06-06T08:26:56.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.951+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ec112c00 con 0x7f92ec0fd150 2024-06-06T08:26:56.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.951+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ec0f9de0 con 0x7f92ec0fd150 2024-06-06T08:26:56.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.952+0000 7f92e97fa640 1 -- 172.21.15.86:0/4260271982 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f92d4030020 con 0x7f92ec0fd150 2024-06-06T08:26:56.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.952+0000 7f92e97fa640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f92d8065430 0x7f92d80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:56.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.952+0000 7f92e97fa640 1 -- 172.21.15.86:0/4260271982 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f92d40a2250 con 0x7f92ec0fd150 2024-06-06T08:26:56.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.952+0000 7f92ebfff640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f92d8065430 0x7f92d80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:56.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.953+0000 7f92ebfff640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f92d8065430 0x7f92d80678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0fae40 tx=0x7f92dc00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:56.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:56.957+0000 7f92e97fa640 1 -- 172.21.15.86:0/4260271982 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f92d4070830 con 0x7f92ec0fd150 2024-06-06T08:26:57.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.180+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f92ec04b480 con 0x7f92ec0fd150 2024-06-06T08:26:57.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.184+0000 7f92e97fa640 1 -- 172.21.15.86:0/4260271982 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f92d4073e50 con 0x7f92ec0fd150 2024-06-06T08:26:57.184 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2024-06-06T08:26:57.184 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDRcmFmS5vrChAArGvJrVDdgYL19fsQF06mWA== 2024-06-06T08:26:57.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.186+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f92d8065430 msgr2=0x7f92d80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.186+0000 7f92f293f640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f92d8065430 0x7f92d80678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0fae40 tx=0x7f92dc00b040 comp rx=0 tx=0).stop 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.186+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 msgr2=0x7f92ec1121b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.186+0000 7f92f293f640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 0x7f92ec1121b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009f80 tx=0x7f92d4009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.186+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 shutdown_connections 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.186+0000 7f92f293f640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f92d8065430 0x7f92d80678e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.186+0000 7f92f293f640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92ec0fd150 0x7f92ec1121b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.186+0000 7f92f293f640 1 --2- 172.21.15.86:0/4260271982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92ec0fc800 0x7f92ec10fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.186+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 >> 172.21.15.86:0/4260271982 conn(0x7f92ec0f6120 msgr2=0x7f92ec0f7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.187+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 shutdown_connections 2024-06-06T08:26:57.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:26:57.187+0000 7f92f293f640 1 -- 172.21.15.86:0/4260271982 wait complete. 2024-06-06T08:26:57.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:57 smithi086 ceph-mon[29114]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:57.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:57 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/4260271982' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T08:26:57.835 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:57 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/4260271982' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T08:26:57.963 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-06T08:26:57.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-06T08:26:57.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-06T08:26:58.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-06T08:26:58.033 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:57 smithi154 ceph-mon[35428]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:58.033 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:57 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/4260271982' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T08:26:58.033 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:57 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/4260271982' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T08:26:58.267 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi154/config 2024-06-06T08:26:59.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.737+0000 7fb9ff718640 1 -- 172.21.15.154:0/3560102285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 msgr2=0x7fb9f80fdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:59.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.737+0000 7fb9ff718640 1 --2- 172.21.15.154:0/3560102285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 0x7fb9f80fdd30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40048d0 tx=0x7fb9f4009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:59.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.737+0000 7fb9ff718640 1 -- 172.21.15.154:0/3560102285 shutdown_connections 2024-06-06T08:26:59.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.737+0000 7fb9ff718640 1 --2- 172.21.15.154:0/3560102285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9f80fe270 0x7fb9f80fa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:59.738 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.737+0000 7fb9ff718640 1 --2- 172.21.15.154:0/3560102285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 0x7fb9f80fdd30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.737+0000 7fb9ff718640 1 -- 172.21.15.154:0/3560102285 >> 172.21.15.154:0/3560102285 conn(0x7fb9f80f6100 msgr2=0x7fb9f80f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.738+0000 7fb9ff718640 1 -- 172.21.15.154:0/3560102285 shutdown_connections 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.738+0000 7fb9ff718640 1 -- 172.21.15.154:0/3560102285 wait complete. 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.738+0000 7fb9ff718640 1 Processor -- start 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.738+0000 7fb9ff718640 1 -- start start 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.738+0000 7fb9ff718640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 0x7fb9f810cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.738+0000 7fb9ff718640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9f80fe270 0x7fb9f810d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.738+0000 7fb9ff718640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f810daa0 con 0x7fb9f80fe270 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.738+0000 7fb9ff718640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f810e2c0 con 0x7fb9f80fd920 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9fd4b4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 0x7fb9f810cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9fd4b4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 0x7fb9f810cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50368/0 (socket says 172.21.15.154:50368) 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9fd4b4640 1 -- 172.21.15.154:0/3277542159 learned_addr learned my addr 172.21.15.154:0/3277542159 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-06-06T08:26:59.739 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9fd4b4640 1 -- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9f80fe270 msgr2=0x7fb9f810d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:59.740 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9fccb3640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9f80fe270 0x7fb9f810d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:59.740 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9fd4b4640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9f80fe270 0x7fb9f810d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:59.740 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9fd4b4640 1 -- 172.21.15.154:0/3277542159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f4005040 con 0x7fb9f80fd920 2024-06-06T08:26:59.740 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9fd4b4640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 0x7fb9f810cf10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009f80 tx=0x7fb9f4009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:59.740 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9edffb640 1 -- 172.21.15.154:0/3277542159 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f4038070 con 0x7fb9f80fd920 2024-06-06T08:26:59.740 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9edffb640 1 -- 172.21.15.154:0/3277542159 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9f4032970 con 0x7fb9f80fd920 2024-06-06T08:26:59.740 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f810e4f0 con 0x7fb9f80fd920 2024-06-06T08:26:59.740 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9edffb640 1 -- 172.21.15.154:0/3277542159 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f4033860 con 0x7fb9f80fd920 2024-06-06T08:26:59.741 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.739+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f810ea30 con 0x7fb9f80fd920 2024-06-06T08:26:59.741 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.740+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f80f9dc0 con 0x7fb9f80fd920 2024-06-06T08:26:59.741 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.741+0000 7fb9edffb640 1 -- 172.21.15.154:0/3277542159 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fb9f4030020 con 0x7fb9f80fd920 2024-06-06T08:26:59.741 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.741+0000 7fb9edffb640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb9e4065430 0x7fb9e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:26:59.741 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.741+0000 7fb9edffb640 1 -- 172.21.15.154:0/3277542159 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb9f40a23c0 con 0x7fb9f80fd920 2024-06-06T08:26:59.741 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.741+0000 7fb9fccb3640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb9e4065430 0x7fb9e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:26:59.741 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.742+0000 7fb9fccb3640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb9e4065430 0x7fb9e40678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb9f810ccc0 tx=0x7fb9e800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:26:59.744 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.745+0000 7fb9edffb640 1 -- 172.21.15.154:0/3277542159 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb9f4070870 con 0x7fb9f80fd920 2024-06-06T08:26:59.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:26:59 smithi154 ceph-mon[35428]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:26:59.974 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.975+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fb9f804b480 con 0x7fb9f80fd920 2024-06-06T08:26:59.979 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.980+0000 7fb9edffb640 1 -- 172.21.15.154:0/3277542159 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fb9f4073e90 con 0x7fb9f80fd920 2024-06-06T08:26:59.979 INFO:teuthology.orchestra.run.smithi154.stdout:[client.1] 2024-06-06T08:26:59.979 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDTcmFmXwNFOhAAmJdBnr6qQpyyD2fcvqZUiA== 2024-06-06T08:26:59.981 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb9e4065430 msgr2=0x7fb9e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:59.981 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb9e4065430 0x7fb9e40678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb9f810ccc0 tx=0x7fb9e800b040 comp rx=0 tx=0).stop 2024-06-06T08:26:59.981 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 msgr2=0x7fb9f810cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:26:59.981 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 0x7fb9f810cf10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009f80 tx=0x7fb9f4009da0 comp rx=0 tx=0).stop 2024-06-06T08:26:59.981 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 shutdown_connections 2024-06-06T08:26:59.982 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb9e4065430 0x7fb9e40678e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:59.982 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9f80fe270 0x7fb9f810d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:59.982 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 --2- 172.21.15.154:0/3277542159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f80fd920 0x7fb9f810cf10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:26:59.982 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 >> 172.21.15.154:0/3277542159 conn(0x7fb9f80f6100 msgr2=0x7fb9f80f77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:26:59.982 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 shutdown_connections 2024-06-06T08:26:59.982 INFO:teuthology.orchestra.run.smithi154.stderr:2024-06-06T08:26:59.982+0000 7fb9ff718640 1 -- 172.21.15.154:0/3277542159 wait complete. 2024-06-06T08:27:00.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:26:59 smithi086 ceph-mon[29114]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:00.554 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-06-06T08:27:00.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-06T08:27:00.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-06T08:27:00.632 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-06T08:27:00.632 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-06T08:27:00.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-06-06T08:27:00.896 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:01.019 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:00 smithi086 ceph-mon[29114]: from='client.? 172.21.15.154:0/3277542159' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T08:27:01.019 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:00 smithi086 ceph-mon[29114]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T08:27:01.019 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:00 smithi086 ceph-mon[29114]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T08:27:01.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:00 smithi154 ceph-mon[35428]: from='client.? 172.21.15.154:0/3277542159' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T08:27:01.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:00 smithi154 ceph-mon[35428]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T08:27:01.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:00 smithi154 ceph-mon[35428]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T08:27:02.233 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:01 smithi086 ceph-mon[29114]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:02.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:01 smithi154 ceph-mon[35428]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:02.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.582+0000 7f2fed850640 1 -- 172.21.15.86:0/2532520059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 msgr2=0x7f2fe80fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:02.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.582+0000 7f2fed850640 1 --2- 172.21.15.86:0/2532520059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 0x7f2fe80fb200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00048d0 tx=0x7f2fd0009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:02.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.582+0000 7f2fed850640 1 -- 172.21.15.86:0/2532520059 shutdown_connections 2024-06-06T08:27:02.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.582+0000 7f2fed850640 1 --2- 172.21.15.86:0/2532520059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 0x7f2fe80fb200 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:02.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.582+0000 7f2fed850640 1 --2- 172.21.15.86:0/2532520059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fe81011d0 0x7f2fe81015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:02.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.582+0000 7f2fed850640 1 -- 172.21.15.86:0/2532520059 >> 172.21.15.86:0/2532520059 conn(0x7f2fe80f6dc0 msgr2=0x7f2fe80f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.582+0000 7f2fed850640 1 -- 172.21.15.86:0/2532520059 shutdown_connections 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.582+0000 7f2fed850640 1 -- 172.21.15.86:0/2532520059 wait complete. 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fed850640 1 Processor -- start 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fed850640 1 -- start start 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fed850640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fe81011d0 0x7f2fe8111f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fed850640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 0x7f2fe8112440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fed850640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe8112a40 con 0x7f2fe8101b70 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fed850640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe810be80 con 0x7f2fe81011d0 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fde7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 0x7f2fe8112440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fde7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 0x7f2fe8112440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:32880/0 (socket says 172.21.15.86:32880) 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fde7fc640 1 -- 172.21.15.86:0/1888204195 learned_addr learned my addr 172.21.15.86:0/1888204195 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fde7fc640 1 -- 172.21.15.86:0/1888204195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fe81011d0 msgr2=0x7f2fe8111f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:02.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fde7fc640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fe81011d0 0x7f2fe8111f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.583+0000 7f2fde7fc640 1 -- 172.21.15.86:0/1888204195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd0005040 con 0x7f2fe8101b70 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.584+0000 7f2fde7fc640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 0x7f2fe8112440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0003080 tx=0x7f2fd0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.584+0000 7f2fdffff640 1 -- 172.21.15.86:0/1888204195 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fd0038070 con 0x7f2fe8101b70 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.584+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe810c100 con 0x7f2fe8101b70 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.584+0000 7f2fdffff640 1 -- 172.21.15.86:0/1888204195 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fd0008ee0 con 0x7f2fe8101b70 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.584+0000 7f2fdffff640 1 -- 172.21.15.86:0/1888204195 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fd003d870 con 0x7f2fe8101b70 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.584+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe810c640 con 0x7f2fe8101b70 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.585+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe80fc940 con 0x7f2fe8101b70 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.585+0000 7f2fdffff640 1 -- 172.21.15.86:0/1888204195 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f2fd0030020 con 0x7f2fe8101b70 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.586+0000 7f2fdffff640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2fd4065430 0x7f2fd40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.586+0000 7f2fdffff640 1 -- 172.21.15.86:0/1888204195 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2fd00a2240 con 0x7f2fe8101b70 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.586+0000 7f2fe6ffd640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2fd4065430 0x7f2fd40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:02.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.586+0000 7f2fe6ffd640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2fd4065430 0x7f2fd40678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8004a50 tx=0x7f2fd8007ba0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:02.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.590+0000 7f2fdffff640 1 -- 172.21.15.86:0/1888204195 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2fd0070770 con 0x7f2fe8101b70 2024-06-06T08:27:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.796+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f2fe8108310 con 0x7f2fe8101b70 2024-06-06T08:27:02.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.799+0000 7f2fdffff640 1 -- 172.21.15.86:0/1888204195 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+161446 (secure 0 0 0) 0x7f2fd0073d90 con 0x7f2fe8101b70 2024-06-06T08:27:02.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:27:02.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2fd4065430 msgr2=0x7f2fd40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:02.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2fd4065430 0x7f2fd40678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8004a50 tx=0x7f2fd8007ba0 comp rx=0 tx=0).stop 2024-06-06T08:27:02.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 msgr2=0x7f2fe8112440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:02.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 0x7f2fe8112440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0003080 tx=0x7f2fd0009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:02.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 shutdown_connections 2024-06-06T08:27:02.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2fd4065430 0x7f2fd40678e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:02.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8101b70 0x7f2fe8112440 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:02.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 --2- 172.21.15.86:0/1888204195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fe81011d0 0x7f2fe8111f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:02.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 >> 172.21.15.86:0/1888204195 conn(0x7f2fe80f6dc0 msgr2=0x7f2fe80f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:02.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 shutdown_connections 2024-06-06T08:27:02.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:02.802+0000 7f2fed850640 1 -- 172.21.15.86:0/1888204195 wait complete. 2024-06-06T08:27:03.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:02 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1888204195' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-06T08:27:03.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:02 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1888204195' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-06T08:27:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":19,"flags":0,"active_gid":14212,"active_name":"smithi086.mfkgze","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3471926911},{"type":"v1","addr":"172.21.15.86:6801","nonce":3471926911}]},"active_addr":"172.21.15.86:6801/3471926911","active_change":"2024-06-06T08:23:24.624872+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi154.kxiapq","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.86:8443/","prometheus":"http://172.21.15.86:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3883742130}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2052995077}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2524477907}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":613193255}]}]} 2024-06-06T08:27:03.535 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-06T08:27:03.535 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-06T08:27:03.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-06T08:27:03.784 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:04.168 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:03 smithi086 ceph-mon[29114]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:03 smithi154 ceph-mon[35428]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 -- 172.21.15.86:0/862928136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd944102c60 msgr2=0x7fd9440fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 --2- 172.21.15.86:0/862928136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd944102c60 0x7fd9440fc2f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0048d0 tx=0x7fd92c009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 -- 172.21.15.86:0/862928136 shutdown_connections 2024-06-06T08:27:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 --2- 172.21.15.86:0/862928136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd944102c60 0x7fd9440fc2f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 --2- 172.21.15.86:0/862928136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9441022c0 0x7fd944102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 -- 172.21.15.86:0/862928136 >> 172.21.15.86:0/862928136 conn(0x7fd9440f7e50 msgr2=0x7fd9440fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:05.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 -- 172.21.15.86:0/862928136 shutdown_connections 2024-06-06T08:27:05.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 -- 172.21.15.86:0/862928136 wait complete. 2024-06-06T08:27:05.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 Processor -- start 2024-06-06T08:27:05.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.252+0000 7fd94999f640 1 -- start start 2024-06-06T08:27:05.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd94999f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9441022c0 0x7fd9441133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:05.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd94999f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd944102c60 0x7fd944113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:05.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd94999f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd944113f90 con 0x7fd9441022c0 2024-06-06T08:27:05.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd94999f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94410d340 con 0x7fd944102c60 2024-06-06T08:27:05.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd942ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9441022c0 0x7fd9441133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd942ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9441022c0 0x7fd9441133c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:32900/0 (socket says 172.21.15.86:32900) 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd942ffd640 1 -- 172.21.15.86:0/977991642 learned_addr learned my addr 172.21.15.86:0/977991642 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd9427fc640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd944102c60 0x7fd944113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd942ffd640 1 -- 172.21.15.86:0/977991642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd944102c60 msgr2=0x7fd944113900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd942ffd640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd944102c60 0x7fd944113900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd942ffd640 1 -- 172.21.15.86:0/977991642 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c005040 con 0x7fd9441022c0 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd942ffd640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9441022c0 0x7fd9441133c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00bbb0 tx=0x7fd93c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd93b7fe640 1 -- 172.21.15.86:0/977991642 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd93c00c9c0 con 0x7fd9441022c0 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd93b7fe640 1 -- 172.21.15.86:0/977991642 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd93c0092e0 con 0x7fd9441022c0 2024-06-06T08:27:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94410d620 con 0x7fd9441022c0 2024-06-06T08:27:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd93b7fe640 1 -- 172.21.15.86:0/977991642 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd93c007640 con 0x7fd9441022c0 2024-06-06T08:27:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.253+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94410db40 con 0x7fd9441022c0 2024-06-06T08:27:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.254+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9440fda30 con 0x7fd9441022c0 2024-06-06T08:27:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.255+0000 7fd93b7fe640 1 -- 172.21.15.86:0/977991642 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd93c012070 con 0x7fd9441022c0 2024-06-06T08:27:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.255+0000 7fd93b7fe640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd930065430 0x7fd9300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.255+0000 7fd93b7fe640 1 -- 172.21.15.86:0/977991642 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd93c082580 con 0x7fd9441022c0 2024-06-06T08:27:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.255+0000 7fd9427fc640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd930065430 0x7fd9300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.256+0000 7fd9427fc640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd930065430 0x7fd9300678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd94410ec80 tx=0x7fd92c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.259+0000 7fd93b7fe640 1 -- 172.21.15.86:0/977991642 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd93c050a30 con 0x7fd9441022c0 2024-06-06T08:27:05.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.425+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd94404b4b0 con 0x7fd9441022c0 2024-06-06T08:27:05.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.426+0000 7fd93b7fe640 1 -- 172.21.15.86:0/977991642 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11081 (secure 0 0 0) 0x7fd93c054050 con 0x7fd9441022c0 2024-06-06T08:27:05.425 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:27:05.425 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":40,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","created":"2024-06-06T08:21:26.882664+0000","modified":"2024-06-06T08:26:46.462722+0000","last_up_change":"2024-06-06T08:26:45.455244+0000","last_in_change":"2024-06-06T08:26:26.696925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-06T08:25:32.951773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"50d6c551-bd2c-4826-a0e9-faf49f3759d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6803","nonce":771268483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6805","nonce":771268483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6809","nonce":771268483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6807","nonce":771268483}]},"public_addr":"172.21.15.86:6803/771268483","cluster_addr":"172.21.15.86:6805/771268483","heartbeat_back_addr":"172.21.15.86:6809/771268483","heartbeat_front_addr":"172.21.15.86:6807/771268483","state":["exists","up"]},{"osd":1,"uuid":"6e6e7221-c7a2-48e9-b584-83b541395665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6811","nonce":1628660505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6813","nonce":1628660505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6817","nonce":1628660505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6815","nonce":1628660505}]},"public_addr":"172.21.15.86:6811/1628660505","cluster_addr":"172.21.15.86:6813/1628660505","heartbeat_back_addr":"172.21.15.86:6817/1628660505","heartbeat_front_addr":"172.21.15.86:6815/1628660505","state":["exists","up"]},{"osd":2,"uuid":"d2ad1948-0ddb-4821-b150-f723c3877245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6819","nonce":3769521712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6821","nonce":3769521712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6825","nonce":3769521712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6823","nonce":3769521712}]},"public_addr":"172.21.15.86:6819/3769521712","cluster_addr":"172.21.15.86:6821/3769521712","heartbeat_back_addr":"172.21.15.86:6825/3769521712","heartbeat_front_addr":"172.21.15.86:6823/3769521712","state":["exists","up"]},{"osd":3,"uuid":"197f72f4-3b9e-4f36-a8fe-854cc214c6ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6801","nonce":1068971095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6803","nonce":1068971095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6807","nonce":1068971095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6805","nonce":1068971095}]},"public_addr":"172.21.15.154:6801/1068971095","cluster_addr":"172.21.15.154:6803/1068971095","heartbeat_back_addr":"172.21.15.154:6807/1068971095","heartbeat_front_addr":"172.21.15.154:6805/1068971095","state":["exists","up"]},{"osd":4,"uuid":"3f63fff7-ffe7-457f-a2c0-a5799c83ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6809","nonce":3195305408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6811","nonce":3195305408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6815","nonce":3195305408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6813","nonce":3195305408}]},"public_addr":"172.21.15.154:6809/3195305408","cluster_addr":"172.21.15.154:6811/3195305408","heartbeat_back_addr":"172.21.15.154:6815/3195305408","heartbeat_front_addr":"172.21.15.154:6813/3195305408","state":["exists","up"]},{"osd":5,"uuid":"194e8288-0427-44ad-8b0d-dc8c67aee3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6817","nonce":2218758456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6819","nonce":2218758456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6823","nonce":2218758456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6821","nonce":2218758456}]},"public_addr":"172.21.15.154:6817/2218758456","cluster_addr":"172.21.15.154:6819/2218758456","heartbeat_back_addr":"172.21.15.154:6823/2218758456","heartbeat_front_addr":"172.21.15.154:6821/2218758456","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:24:36.527260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:25:03.405907+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:25:30.733020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:25:54.062475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:26:18.638828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:26:43.487776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2470791934":"2024-06-07T08:23:24.624760+0000","172.21.15.86:6800/2470791934":"2024-06-07T08:23:24.624760+0000","172.21.15.86:6801/3222204828":"2024-06-07T08:21:56.545284+0000","172.21.15.86:6800/3222204828":"2024-06-07T08:21:56.545284+0000","172.21.15.86:0/3370799480":"2024-06-07T08:21:56.545284+0000","172.21.15.86:6801/933334455":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/3574006712":"2024-06-07T08:21:56.545284+0000","172.21.15.86:0/2909599705":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/2894717502":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/2181614567":"2024-06-07T08:21:56.545284+0000","172.21.15.86:6800/933334455":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/808628117":"2024-06-07T08:23:24.624760+0000","172.21.15.86:0/3541586639":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/2450050335":"2024-06-07T08:23:24.624760+0000","172.21.15.86:0/753216971":"2024-06-07T08:23:24.624760+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-06T08:27:05.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd930065430 msgr2=0x7fd9300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:05.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd930065430 0x7fd9300678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd94410ec80 tx=0x7fd92c035040 comp rx=0 tx=0).stop 2024-06-06T08:27:05.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9441022c0 msgr2=0x7fd9441133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:05.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9441022c0 0x7fd9441133c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00bbb0 tx=0x7fd93c009840 comp rx=0 tx=0).stop 2024-06-06T08:27:05.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 shutdown_connections 2024-06-06T08:27:05.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd944102c60 0x7fd944113900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:05.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd930065430 0x7fd9300678e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:05.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 --2- 172.21.15.86:0/977991642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9441022c0 0x7fd9441133c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:05.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 >> 172.21.15.86:0/977991642 conn(0x7fd9440f7e50 msgr2=0x7fd9440f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:05.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 shutdown_connections 2024-06-06T08:27:05.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:05.428+0000 7fd94999f640 1 -- 172.21.15.86:0/977991642 wait complete. 2024-06-06T08:27:06.039 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-06T08:27:06.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-06T08:27:06.294 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:06.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:05 smithi086 ceph-mon[29114]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:06.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:05 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/977991642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T08:27:06.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:05 smithi154 ceph-mon[35428]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:06.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:05 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/977991642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T08:27:07.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 -- 172.21.15.86:0/4048248786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab81022a0 msgr2=0x7fdab8102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 --2- 172.21.15.86:0/4048248786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab81022a0 0x7fdab8102670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80048d0 tx=0x7fdaa8009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 -- 172.21.15.86:0/4048248786 shutdown_connections 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 --2- 172.21.15.86:0/4048248786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab8102c40 0x7fdab80fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 --2- 172.21.15.86:0/4048248786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab81022a0 0x7fdab8102670 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 -- 172.21.15.86:0/4048248786 >> 172.21.15.86:0/4048248786 conn(0x7fdab80f7e50 msgr2=0x7fdab80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 -- 172.21.15.86:0/4048248786 shutdown_connections 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 -- 172.21.15.86:0/4048248786 wait complete. 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 Processor -- start 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.926+0000 7fdabde91640 1 -- start start 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdabde91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab81022a0 0x7fdab806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdabde91640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab8102c40 0x7fdab806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:07.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdabde91640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80701f0 con 0x7fdab8102c40 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdabde91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab8070360 con 0x7fdab81022a0 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdaaffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab8102c40 0x7fdab806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdaaffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab8102c40 0x7fdab806fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60596/0 (socket says 172.21.15.86:60596) 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdaaffff640 1 -- 172.21.15.86:0/486060469 learned_addr learned my addr 172.21.15.86:0/486060469 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdab77fe640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab81022a0 0x7fdab806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdaaffff640 1 -- 172.21.15.86:0/486060469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab81022a0 msgr2=0x7fdab806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdaaffff640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab81022a0 0x7fdab806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdaaffff640 1 -- 172.21.15.86:0/486060469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa8005040 con 0x7fdab8102c40 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdaaffff640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab8102c40 0x7fdab806fcb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0002a50 tx=0x7fdaa0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdab4ff9640 1 -- 172.21.15.86:0/486060469 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaa0007c30 con 0x7fdab8102c40 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdab4ff9640 1 -- 172.21.15.86:0/486060469 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaa000f040 con 0x7fdab8102c40 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab80705f0 con 0x7fdab8102c40 2024-06-06T08:27:07.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.927+0000 7fdab4ff9640 1 -- 172.21.15.86:0/486060469 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaa000db60 con 0x7fdab8102c40 2024-06-06T08:27:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.928+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab81196e0 con 0x7fdab8102c40 2024-06-06T08:27:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.928+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab80fda10 con 0x7fdab8102c40 2024-06-06T08:27:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.929+0000 7fdab4ff9640 1 -- 172.21.15.86:0/486060469 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fdaa000dcc0 con 0x7fdab8102c40 2024-06-06T08:27:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.929+0000 7fdab4ff9640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdaa4065430 0x7fdaa40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.929+0000 7fdab4ff9640 1 -- 172.21.15.86:0/486060469 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdaa0082310 con 0x7fdab8102c40 2024-06-06T08:27:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.929+0000 7fdab77fe640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdaa4065430 0x7fdaa40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.930+0000 7fdab77fe640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdaa4065430 0x7fdaa40678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009f80 tx=0x7fdaa8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:07.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:07.933+0000 7fdab4ff9640 1 -- 172.21.15.86:0/486060469 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdaa00507c0 con 0x7fdab8102c40 2024-06-06T08:27:08.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.102+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdab80fba90 con 0x7fdab8102c40 2024-06-06T08:27:08.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.103+0000 7fdab4ff9640 1 -- 172.21.15.86:0/486060469 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11081 (secure 0 0 0) 0x7fdaa0053de0 con 0x7fdab8102c40 2024-06-06T08:27:08.102 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:27:08.102 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":40,"fsid":"a8550dee-23dd-11ef-bc9b-c7b262605968","created":"2024-06-06T08:21:26.882664+0000","modified":"2024-06-06T08:26:46.462722+0000","last_up_change":"2024-06-06T08:26:45.455244+0000","last_in_change":"2024-06-06T08:26:26.696925+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-06T08:25:32.951773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"50d6c551-bd2c-4826-a0e9-faf49f3759d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6803","nonce":771268483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6805","nonce":771268483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6809","nonce":771268483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":771268483},{"type":"v1","addr":"172.21.15.86:6807","nonce":771268483}]},"public_addr":"172.21.15.86:6803/771268483","cluster_addr":"172.21.15.86:6805/771268483","heartbeat_back_addr":"172.21.15.86:6809/771268483","heartbeat_front_addr":"172.21.15.86:6807/771268483","state":["exists","up"]},{"osd":1,"uuid":"6e6e7221-c7a2-48e9-b584-83b541395665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6811","nonce":1628660505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6813","nonce":1628660505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6817","nonce":1628660505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1628660505},{"type":"v1","addr":"172.21.15.86:6815","nonce":1628660505}]},"public_addr":"172.21.15.86:6811/1628660505","cluster_addr":"172.21.15.86:6813/1628660505","heartbeat_back_addr":"172.21.15.86:6817/1628660505","heartbeat_front_addr":"172.21.15.86:6815/1628660505","state":["exists","up"]},{"osd":2,"uuid":"d2ad1948-0ddb-4821-b150-f723c3877245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6819","nonce":3769521712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6821","nonce":3769521712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6825","nonce":3769521712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3769521712},{"type":"v1","addr":"172.21.15.86:6823","nonce":3769521712}]},"public_addr":"172.21.15.86:6819/3769521712","cluster_addr":"172.21.15.86:6821/3769521712","heartbeat_back_addr":"172.21.15.86:6825/3769521712","heartbeat_front_addr":"172.21.15.86:6823/3769521712","state":["exists","up"]},{"osd":3,"uuid":"197f72f4-3b9e-4f36-a8fe-854cc214c6ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6801","nonce":1068971095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6803","nonce":1068971095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6807","nonce":1068971095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1068971095},{"type":"v1","addr":"172.21.15.154:6805","nonce":1068971095}]},"public_addr":"172.21.15.154:6801/1068971095","cluster_addr":"172.21.15.154:6803/1068971095","heartbeat_back_addr":"172.21.15.154:6807/1068971095","heartbeat_front_addr":"172.21.15.154:6805/1068971095","state":["exists","up"]},{"osd":4,"uuid":"3f63fff7-ffe7-457f-a2c0-a5799c83ba32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6809","nonce":3195305408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6811","nonce":3195305408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6815","nonce":3195305408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3195305408},{"type":"v1","addr":"172.21.15.154:6813","nonce":3195305408}]},"public_addr":"172.21.15.154:6809/3195305408","cluster_addr":"172.21.15.154:6811/3195305408","heartbeat_back_addr":"172.21.15.154:6815/3195305408","heartbeat_front_addr":"172.21.15.154:6813/3195305408","state":["exists","up"]},{"osd":5,"uuid":"194e8288-0427-44ad-8b0d-dc8c67aee3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6817","nonce":2218758456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6819","nonce":2218758456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6823","nonce":2218758456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2218758456},{"type":"v1","addr":"172.21.15.154:6821","nonce":2218758456}]},"public_addr":"172.21.15.154:6817/2218758456","cluster_addr":"172.21.15.154:6819/2218758456","heartbeat_back_addr":"172.21.15.154:6823/2218758456","heartbeat_front_addr":"172.21.15.154:6821/2218758456","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:24:36.527260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:25:03.405907+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:25:30.733020+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:25:54.062475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:26:18.638828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T08:26:43.487776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2470791934":"2024-06-07T08:23:24.624760+0000","172.21.15.86:6800/2470791934":"2024-06-07T08:23:24.624760+0000","172.21.15.86:6801/3222204828":"2024-06-07T08:21:56.545284+0000","172.21.15.86:6800/3222204828":"2024-06-07T08:21:56.545284+0000","172.21.15.86:0/3370799480":"2024-06-07T08:21:56.545284+0000","172.21.15.86:6801/933334455":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/3574006712":"2024-06-07T08:21:56.545284+0000","172.21.15.86:0/2909599705":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/2894717502":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/2181614567":"2024-06-07T08:21:56.545284+0000","172.21.15.86:6800/933334455":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/808628117":"2024-06-07T08:23:24.624760+0000","172.21.15.86:0/3541586639":"2024-06-07T08:22:25.340003+0000","172.21.15.86:0/2450050335":"2024-06-07T08:23:24.624760+0000","172.21.15.86:0/753216971":"2024-06-07T08:23:24.624760+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdaa4065430 msgr2=0x7fdaa40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdaa4065430 0x7fdaa40678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009f80 tx=0x7fdaa8035040 comp rx=0 tx=0).stop 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab8102c40 msgr2=0x7fdab806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab8102c40 0x7fdab806fcb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0002a50 tx=0x7fdaa0004600 comp rx=0 tx=0).stop 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 shutdown_connections 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdaa4065430 0x7fdaa40678e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab8102c40 0x7fdab806fcb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 --2- 172.21.15.86:0/486060469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab81022a0 0x7fdab806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 >> 172.21.15.86:0/486060469 conn(0x7fdab80f7e50 msgr2=0x7fdab80f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 shutdown_connections 2024-06-06T08:27:08.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:08.105+0000 7fdabde91640 1 -- 172.21.15.86:0/486060469 wait complete. 2024-06-06T08:27:08.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:07 smithi086 ceph-mon[29114]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:08.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:07 smithi154 ceph-mon[35428]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:08.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-06T08:27:08.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-06T08:27:08.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-06T08:27:08.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-06T08:27:08.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-06T08:27:08.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-06T08:27:09.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:08 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/486060469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T08:27:09.112 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:09.118 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:09.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:09 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/486060469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T08:27:10.200 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:09 smithi086 ceph-mon[29114]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:10.200 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:27:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:09 smithi154 ceph-mon[35428]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:27:11.530 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:11.530 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:11.534 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:12.154 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:12 smithi086 ceph-mon[29114]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:12.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:12 smithi154 ceph-mon[35428]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:14.208 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:14 smithi086 ceph-mon[29114]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:14.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:14 smithi154 ceph-mon[35428]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.969+0000 7f169ac45640 1 -- 172.21.15.86:0/2962480446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1694102c40 msgr2=0x7f16940fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.969+0000 7f169ac45640 1 --2- 172.21.15.86:0/2962480446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1694102c40 0x7f16940fc2d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f16800048d0 tx=0x7f1680009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.970+0000 7f169ac45640 1 -- 172.21.15.86:0/2962480446 shutdown_connections 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.970+0000 7f169ac45640 1 --2- 172.21.15.86:0/2962480446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1694102c40 0x7f16940fc2d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.970+0000 7f169ac45640 1 --2- 172.21.15.86:0/2962480446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16941022a0 0x7f1694102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.970+0000 7f169ac45640 1 -- 172.21.15.86:0/2962480446 >> 172.21.15.86:0/2962480446 conn(0x7f16940f7e50 msgr2=0x7f16940fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.970+0000 7f169ac45640 1 -- 172.21.15.86:0/2962480446 shutdown_connections 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.970+0000 7f169ac45640 1 -- 172.21.15.86:0/2962480446 wait complete. 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.970+0000 7f169ac45640 1 Processor -- start 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.970+0000 7f169ac45640 1 -- start start 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f169ac45640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16941022a0 0x7f16941185a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f169ac45640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1694102c40 0x7f1694118b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f169ac45640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16941191e0 con 0x7f16941022a0 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f169ac45640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169410ae50 con 0x7f1694102c40 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f16989e1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16941022a0 0x7f16941185a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f16989e1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16941022a0 0x7f16941185a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60618/0 (socket says 172.21.15.86:60618) 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f16989e1640 1 -- 172.21.15.86:0/2974490536 learned_addr learned my addr 172.21.15.86:0/2974490536 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f1693fff640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1694102c40 0x7f1694118b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f16989e1640 1 -- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1694102c40 msgr2=0x7f1694118b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f16989e1640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1694102c40 0x7f1694118b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f16989e1640 1 -- 172.21.15.86:0/2974490536 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680005040 con 0x7f16941022a0 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f16989e1640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16941022a0 0x7f16941185a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f168800b790 tx=0x7f16880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f16917fa640 1 -- 172.21.15.86:0/2974490536 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f168800c700 con 0x7f16941022a0 2024-06-06T08:27:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.971+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169410b130 con 0x7f16941022a0 2024-06-06T08:27:15.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.972+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169410b650 con 0x7f16941022a0 2024-06-06T08:27:15.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.972+0000 7f16917fa640 1 -- 172.21.15.86:0/2974490536 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f168800cd30 con 0x7f16941022a0 2024-06-06T08:27:15.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.972+0000 7f16917fa640 1 -- 172.21.15.86:0/2974490536 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1688013370 con 0x7f16941022a0 2024-06-06T08:27:15.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.972+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1678000fd0 con 0x7f16941022a0 2024-06-06T08:27:15.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.973+0000 7f16917fa640 1 -- 172.21.15.86:0/2974490536 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f1688012070 con 0x7f16941022a0 2024-06-06T08:27:15.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.973+0000 7f16917fa640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1684065430 0x7f16840678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:15.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.974+0000 7f16917fa640 1 -- 172.21.15.86:0/2974490536 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1688082350 con 0x7f16941022a0 2024-06-06T08:27:15.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.974+0000 7f1693fff640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1684065430 0x7f16840678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:15.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.974+0000 7f16917fa640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] conn(0x7f168406b1b0 0x7f168406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:15.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.974+0000 7f16917fa640 1 -- 172.21.15.86:0/2974490536 --> [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168406dc70 con 0x7f168406b1b0 2024-06-06T08:27:15.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.974+0000 7f16917fa640 1 -- 172.21.15.86:0/2974490536 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1688082730 con 0x7f16941022a0 2024-06-06T08:27:15.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.974+0000 7f16991e2640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] conn(0x7f168406b1b0 0x7f168406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:15.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.974+0000 7f1693fff640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1684065430 0x7f16840678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f169410c740 tx=0x7f1680035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:15.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.974+0000 7f16991e2640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] conn(0x7f168406b1b0 0x7f168406d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:15.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.975+0000 7f16917fa640 1 -- 172.21.15.86:0/2974490536 <== osd.5 v2:172.21.15.154:6816/2218758456 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f168406dc70 con 0x7f168406b1b0 2024-06-06T08:27:15.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.989+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 --> [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1678002ca0 con 0x7f168406b1b0 2024-06-06T08:27:15.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.989+0000 7f16917fa640 1 -- 172.21.15.86:0/2974490536 <== osd.5 v2:172.21.15.154:6816/2218758456 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1678002ca0 con 0x7f168406b1b0 2024-06-06T08:27:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.989+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] conn(0x7f168406b1b0 msgr2=0x7f168406d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.989+0000 7f169ac45640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] conn(0x7f168406b1b0 0x7f168406d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1684065430 msgr2=0x7f16840678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1684065430 0x7f16840678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f169410c740 tx=0x7f1680035040 comp rx=0 tx=0).stop 2024-06-06T08:27:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16941022a0 msgr2=0x7f16941185a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16941022a0 0x7f16941185a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f168800b790 tx=0x7f16880091d0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 shutdown_connections 2024-06-06T08:27:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:6816/2218758456,v1:172.21.15.154:6817/2218758456] conn(0x7f168406b1b0 0x7f168406d5c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1694102c40 0x7f1694118b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f1684065430 0x7f16840678e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 --2- 172.21.15.86:0/2974490536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16941022a0 0x7f16941185a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 >> 172.21.15.86:0/2974490536 conn(0x7f16940f7e50 msgr2=0x7f1694105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 shutdown_connections 2024-06-06T08:27:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:15.990+0000 7f169ac45640 1 -- 172.21.15.86:0/2974490536 wait complete. 2024-06-06T08:27:16.078 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:16 smithi086 ceph-mon[29114]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:16.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:16 smithi154 ceph-mon[35428]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:16.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.620+0000 7f03b2bc2640 1 -- 172.21.15.86:0/4222616714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac1011b0 msgr2=0x7f03ac101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:16.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.620+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/4222616714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac1011b0 0x7f03ac101580 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f03a00048d0 tx=0x7f03a0009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.620+0000 7f03b2bc2640 1 -- 172.21.15.86:0/4222616714 shutdown_connections 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.620+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/4222616714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ac101b50 0x7f03ac0fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.620+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/4222616714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac1011b0 0x7f03ac101580 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.620+0000 7f03b2bc2640 1 -- 172.21.15.86:0/4222616714 >> 172.21.15.86:0/4222616714 conn(0x7f03ac0f6d60 msgr2=0x7f03ac0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.620+0000 7f03b2bc2640 1 -- 172.21.15.86:0/4222616714 shutdown_connections 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.620+0000 7f03b2bc2640 1 -- 172.21.15.86:0/4222616714 wait complete. 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.621+0000 7f03b2bc2640 1 Processor -- start 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.621+0000 7f03b2bc2640 1 -- start start 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.621+0000 7f03b2bc2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ac1011b0 0x7f03ac10be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.621+0000 7f03b2bc2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac101b50 0x7f03ac10c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.621+0000 7f03b2bc2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac10c920 con 0x7f03ac101b50 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.621+0000 7f03b2bc2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac10ca60 con 0x7f03ac1011b0 2024-06-06T08:27:16.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.621+0000 7f03abfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac101b50 0x7f03ac10c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.621+0000 7f03abfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac101b50 0x7f03ac10c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60628/0 (socket says 172.21.15.86:60628) 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.621+0000 7f03abfff640 1 -- 172.21.15.86:0/3004040186 learned_addr learned my addr 172.21.15.86:0/3004040186 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03b095e640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ac1011b0 0x7f03ac10be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03abfff640 1 -- 172.21.15.86:0/3004040186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ac1011b0 msgr2=0x7f03ac10be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03abfff640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ac1011b0 0x7f03ac10be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03abfff640 1 -- 172.21.15.86:0/3004040186 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a0005040 con 0x7f03ac101b50 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03b095e640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ac1011b0 0x7f03ac10be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03abfff640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac101b50 0x7f03ac10c350 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f039800b790 tx=0x7f03980091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03a97fa640 1 -- 172.21.15.86:0/3004040186 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f039800c700 con 0x7f03ac101b50 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ac119480 con 0x7f03ac101b50 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03a97fa640 1 -- 172.21.15.86:0/3004040186 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f039800cd30 con 0x7f03ac101b50 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03a97fa640 1 -- 172.21.15.86:0/3004040186 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0398013370 con 0x7f03ac101b50 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.622+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ac119860 con 0x7f03ac101b50 2024-06-06T08:27:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.623+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f03ac0f8720 con 0x7f03ac101b50 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.624+0000 7f03a97fa640 1 -- 172.21.15.86:0/3004040186 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0398012070 con 0x7f03ac101b50 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.624+0000 7f03a97fa640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f039c065430 0x7f039c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.624+0000 7f03a97fa640 1 -- 172.21.15.86:0/3004040186 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0398082080 con 0x7f03ac101b50 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.624+0000 7f03b095e640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f039c065430 0x7f039c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.624+0000 7f03a97fa640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] conn(0x7f039c06b1b0 0x7f039c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.624+0000 7f03a97fa640 1 -- 172.21.15.86:0/3004040186 --> [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f039c06dc70 con 0x7f039c06b1b0 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.624+0000 7f03a97fa640 1 -- 172.21.15.86:0/3004040186 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03980846d0 con 0x7f03ac101b50 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.624+0000 7f03b115f640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] conn(0x7f039c06b1b0 0x7f039c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.625+0000 7f03b095e640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f039c065430 0x7f039c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009f80 tx=0x7f03a0009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.625+0000 7f03b115f640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] conn(0x7f039c06b1b0 0x7f039c06d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:16.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.625+0000 7f03a97fa640 1 -- 172.21.15.86:0/3004040186 <== osd.2 v2:172.21.15.86:6818/3769521712 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f039c06dc70 con 0x7f039c06b1b0 2024-06-06T08:27:16.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.638+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 --> [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f03ac0fc920 con 0x7f039c06b1b0 2024-06-06T08:27:16.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.638+0000 7f03a97fa640 1 -- 172.21.15.86:0/3004040186 <== osd.2 v2:172.21.15.86:6818/3769521712 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f03ac0fc920 con 0x7f039c06b1b0 2024-06-06T08:27:16.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] conn(0x7f039c06b1b0 msgr2=0x7f039c06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:16.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] conn(0x7f039c06b1b0 0x7f039c06d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f039c065430 msgr2=0x7f039c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:16.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f039c065430 0x7f039c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009f80 tx=0x7f03a0009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac101b50 msgr2=0x7f03ac10c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:16.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac101b50 0x7f03ac10c350 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f039800b790 tx=0x7f03980091d0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 shutdown_connections 2024-06-06T08:27:16.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6818/3769521712,v1:172.21.15.86:6819/3769521712] conn(0x7f039c06b1b0 0x7f039c06d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f039c065430 0x7f039c0678e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03ac101b50 0x7f03ac10c350 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 --2- 172.21.15.86:0/3004040186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ac1011b0 0x7f03ac10be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:16.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 >> 172.21.15.86:0/3004040186 conn(0x7f03ac0f6d60 msgr2=0x7f03ac0f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:16.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 shutdown_connections 2024-06-06T08:27:16.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:16.639+0000 7f03b2bc2640 1 -- 172.21.15.86:0/3004040186 wait complete. 2024-06-06T08:27:17.296 INFO:teuthology.orchestra.run.smithi086.stdout:167503724553 2024-06-06T08:27:17.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-06T08:27:17.888 INFO:teuthology.orchestra.run.smithi086.stdout:81604378648 2024-06-06T08:27:17.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-06T08:27:18.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.098+0000 7fa511dd6640 1 -- 172.21.15.86:0/2590747273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 msgr2=0x7fa50c1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:18.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.098+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2590747273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 0x7fa50c1015a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0048d0 tx=0x7fa4fc009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.098+0000 7fa511dd6640 1 -- 172.21.15.86:0/2590747273 shutdown_connections 2024-06-06T08:27:18.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.098+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2590747273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa50c101b70 0x7fa50c0fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.098+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2590747273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 0x7fa50c1015a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.098+0000 7fa511dd6640 1 -- 172.21.15.86:0/2590747273 >> 172.21.15.86:0/2590747273 conn(0x7fa50c0f6d80 msgr2=0x7fa50c0f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:18.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.098+0000 7fa511dd6640 1 -- 172.21.15.86:0/2590747273 shutdown_connections 2024-06-06T08:27:18.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.098+0000 7fa511dd6640 1 -- 172.21.15.86:0/2590747273 wait complete. 2024-06-06T08:27:18.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.099+0000 7fa511dd6640 1 Processor -- start 2024-06-06T08:27:18.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.099+0000 7fa511dd6640 1 -- start start 2024-06-06T08:27:18.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa511dd6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 0x7fa50c113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:18.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa511dd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa50c101b70 0x7fa50c113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:18.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa511dd6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c113bf0 con 0x7fa50c1011d0 2024-06-06T08:27:18.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa511dd6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c10cfa0 con 0x7fa50c101b70 2024-06-06T08:27:18.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa50b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 0x7fa50c113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:18.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa50b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 0x7fa50c113020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35970/0 (socket says 172.21.15.86:35970) 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa50b7fe640 1 -- 172.21.15.86:0/2170203312 learned_addr learned my addr 172.21.15.86:0/2170203312 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa50affd640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa50c101b70 0x7fa50c113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa50b7fe640 1 -- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa50c101b70 msgr2=0x7fa50c113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa50b7fe640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa50c101b70 0x7fa50c113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa50b7fe640 1 -- 172.21.15.86:0/2170203312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc005040 con 0x7fa50c1011d0 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa50b7fe640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 0x7fa50c113020 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc003080 tx=0x7fa4fc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa4f3fff640 1 -- 172.21.15.86:0/2170203312 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4fc038070 con 0x7fa50c1011d0 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50c10d220 con 0x7fa50c1011d0 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa4f3fff640 1 -- 172.21.15.86:0/2170203312 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4fc008ee0 con 0x7fa50c1011d0 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa4f3fff640 1 -- 172.21.15.86:0/2170203312 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4fc03d870 con 0x7fa50c1011d0 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.100+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50c10d6e0 con 0x7fa50c1011d0 2024-06-06T08:27:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.101+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa50c10dbc0 con 0x7fa50c1011d0 2024-06-06T08:27:18.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.102+0000 7fa4f3fff640 1 -- 172.21.15.86:0/2170203312 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa4fc030020 con 0x7fa50c1011d0 2024-06-06T08:27:18.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.102+0000 7fa4f3fff640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa4f4065430 0x7fa4f40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:18.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.102+0000 7fa4f3fff640 1 -- 172.21.15.86:0/2170203312 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa4fc0a21a0 con 0x7fa50c1011d0 2024-06-06T08:27:18.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.103+0000 7fa50affd640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa4f4065430 0x7fa4f40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:18.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.103+0000 7fa4f3fff640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] conn(0x7fa4f406b1b0 0x7fa4f406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:18.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.103+0000 7fa4f3fff640 1 -- 172.21.15.86:0/2170203312 --> [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4f406dc70 con 0x7fa4f406b1b0 2024-06-06T08:27:18.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.103+0000 7fa4f3fff640 1 -- 172.21.15.86:0/2170203312 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4fc0a4790 con 0x7fa50c1011d0 2024-06-06T08:27:18.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.103+0000 7fa50affd640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa4f4065430 0x7fa4f40678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa50c10e8e0 tx=0x7fa500009120 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:18.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.103+0000 7fa50bfff640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] conn(0x7fa4f406b1b0 0x7fa4f406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:18.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.104+0000 7fa50bfff640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] conn(0x7fa4f406b1b0 0x7fa4f406d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:18.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.104+0000 7fa4f3fff640 1 -- 172.21.15.86:0/2170203312 <== osd.3 v2:172.21.15.154:6800/1068971095 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa4f406dc70 con 0x7fa4f406b1b0 2024-06-06T08:27:18.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.118+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 --> [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa50c0fc940 con 0x7fa4f406b1b0 2024-06-06T08:27:18.118 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:18 smithi086 ceph-mon[29114]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:18.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa4f3fff640 1 -- 172.21.15.86:0/2170203312 <== osd.3 v2:172.21.15.154:6800/1068971095 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa50c0fc940 con 0x7fa4f406b1b0 2024-06-06T08:27:18.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] conn(0x7fa4f406b1b0 msgr2=0x7fa4f406d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:18.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] conn(0x7fa4f406b1b0 0x7fa4f406d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa4f4065430 msgr2=0x7fa4f40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa4f4065430 0x7fa4f40678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa50c10e8e0 tx=0x7fa500009120 comp rx=0 tx=0).stop 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 msgr2=0x7fa50c113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 0x7fa50c113020 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc003080 tx=0x7fa4fc009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 shutdown_connections 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:6800/1068971095,v1:172.21.15.154:6801/1068971095] conn(0x7fa4f406b1b0 0x7fa4f406d5c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa50c101b70 0x7fa50c113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa4f4065430 0x7fa4f40678e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 --2- 172.21.15.86:0/2170203312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c1011d0 0x7fa50c113020 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 >> 172.21.15.86:0/2170203312 conn(0x7fa50c0f6d80 msgr2=0x7fa50c0f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 shutdown_connections 2024-06-06T08:27:18.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:18.119+0000 7fa511dd6640 1 -- 172.21.15.86:0/2170203312 wait complete. 2024-06-06T08:27:18.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:18 smithi154 ceph-mon[35428]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:18.831 INFO:teuthology.orchestra.run.smithi086.stdout:111669149715 2024-06-06T08:27:18.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-06T08:27:19.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.028+0000 7f68c305a640 1 -- 172.21.15.86:0/280305278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fead0 msgr2=0x7f68bc0fa940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.028+0000 7f68c305a640 1 --2- 172.21.15.86:0/280305278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fead0 0x7f68bc0fa940 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f68a80048d0 tx=0x7f68a8009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.028+0000 7f68c305a640 1 -- 172.21.15.86:0/280305278 shutdown_connections 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.028+0000 7f68c305a640 1 --2- 172.21.15.86:0/280305278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fead0 0x7f68bc0fa940 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.028+0000 7f68c305a640 1 --2- 172.21.15.86:0/280305278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68bc0fe1c0 0x7f68bc0fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.028+0000 7f68c305a640 1 -- 172.21.15.86:0/280305278 >> 172.21.15.86:0/280305278 conn(0x7f68bc071860 msgr2=0x7f68bc071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.028+0000 7f68c305a640 1 -- 172.21.15.86:0/280305278 shutdown_connections 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.028+0000 7f68c305a640 1 -- 172.21.15.86:0/280305278 wait complete. 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68c305a640 1 Processor -- start 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68c305a640 1 -- start start 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68c305a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fe1c0 0x7f68bc0fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68c305a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68bc0fead0 0x7f68bc0fc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68c305a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0fc5a0 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68c305a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0fc710 con 0x7f68bc0fead0 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68c0df6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fe1c0 0x7f68bc0fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68c0df6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fe1c0 0x7f68bc0fbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35974/0 (socket says 172.21.15.86:35974) 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68c0df6640 1 -- 172.21.15.86:0/2449319002 learned_addr learned my addr 172.21.15.86:0/2449319002 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.029+0000 7f68bbfff640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68bc0fead0 0x7f68bc0fc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68c0df6640 1 -- 172.21.15.86:0/2449319002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68bc0fead0 msgr2=0x7f68bc0fc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68c0df6640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68bc0fead0 0x7f68bc0fc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68c0df6640 1 -- 172.21.15.86:0/2449319002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8005040 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68bbfff640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68bc0fead0 0x7f68bc0fc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68c0df6640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fe1c0 0x7f68bc0fbb20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f68b0002a50 tx=0x7f68b0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68b97fa640 1 -- 172.21.15.86:0/2449319002 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b0007c30 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68bc0fc9a0 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68bc119740 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68b97fa640 1 -- 172.21.15.86:0/2449319002 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68b000f040 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.030+0000 7f68b97fa640 1 -- 172.21.15.86:0/2449319002 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b000dad0 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.031+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68bc0f9df0 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.031+0000 7f68b97fa640 1 -- 172.21.15.86:0/2449319002 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f68b0009730 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.032+0000 7f68b97fa640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f68ac065430 0x7f68ac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:19.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.032+0000 7f68bbfff640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f68ac065430 0x7f68ac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:19.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.032+0000 7f68b97fa640 1 -- 172.21.15.86:0/2449319002 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f68b0081d50 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.032+0000 7f68b97fa640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] conn(0x7f68ac06b1b0 0x7f68ac06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.032+0000 7f68b97fa640 1 -- 172.21.15.86:0/2449319002 --> [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68ac06dc70 con 0x7f68ac06b1b0 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.032+0000 7f68b97fa640 1 -- 172.21.15.86:0/2449319002 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68b00873b0 con 0x7f68bc0fe1c0 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.032+0000 7f68c15f7640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] conn(0x7f68ac06b1b0 0x7f68ac06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.032+0000 7f68bbfff640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f68ac065430 0x7f68ac0678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68a8000c00 tx=0x7f68a8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.032+0000 7f68c15f7640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] conn(0x7f68ac06b1b0 0x7f68ac06d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.033+0000 7f68b97fa640 1 -- 172.21.15.86:0/2449319002 <== osd.0 v2:172.21.15.86:6802/771268483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f68ac06dc70 con 0x7f68ac06b1b0 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.046+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 --> [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f68bc064ab0 con 0x7f68ac06b1b0 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.046+0000 7f68b97fa640 1 -- 172.21.15.86:0/2449319002 <== osd.0 v2:172.21.15.86:6802/771268483 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f68bc064ab0 con 0x7f68ac06b1b0 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.046+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] conn(0x7f68ac06b1b0 msgr2=0x7f68ac06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.046+0000 7f68c305a640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] conn(0x7f68ac06b1b0 0x7f68ac06d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.046+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f68ac065430 msgr2=0x7f68ac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.046+0000 7f68c305a640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f68ac065430 0x7f68ac0678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68a8000c00 tx=0x7f68a8035040 comp rx=0 tx=0).stop 2024-06-06T08:27:19.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.046+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fe1c0 msgr2=0x7f68bc0fbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.046+0000 7f68c305a640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fe1c0 0x7f68bc0fbb20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f68b0002a50 tx=0x7f68b0004600 comp rx=0 tx=0).stop 2024-06-06T08:27:19.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.047+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 shutdown_connections 2024-06-06T08:27:19.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.047+0000 7f68c305a640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6802/771268483,v1:172.21.15.86:6803/771268483] conn(0x7f68ac06b1b0 0x7f68ac06d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.047+0000 7f68c305a640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68bc0fead0 0x7f68bc0fc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.047+0000 7f68c305a640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f68ac065430 0x7f68ac0678e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.047+0000 7f68c305a640 1 --2- 172.21.15.86:0/2449319002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68bc0fe1c0 0x7f68bc0fbb20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.047+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 >> 172.21.15.86:0/2449319002 conn(0x7f68bc071860 msgr2=0x7f68bc0f93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:19.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.047+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 shutdown_connections 2024-06-06T08:27:19.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.047+0000 7f68c305a640 1 -- 172.21.15.86:0/2449319002 wait complete. 2024-06-06T08:27:19.289 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:19.683 INFO:teuthology.orchestra.run.smithi086.stdout:38654705699 2024-06-06T08:27:19.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-06T08:27:19.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.834+0000 7f4c4680e640 1 -- 172.21.15.86:0/83804736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fb570 msgr2=0x7f4c400fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.834+0000 7f4c4680e640 1 --2- 172.21.15.86:0/83804736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fb570 0x7f4c400fb940 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4c300048d0 tx=0x7f4c30009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.834+0000 7f4c4680e640 1 -- 172.21.15.86:0/83804736 shutdown_connections 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.834+0000 7f4c4680e640 1 --2- 172.21.15.86:0/83804736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c400fbe80 0x7f4c400fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.834+0000 7f4c4680e640 1 --2- 172.21.15.86:0/83804736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fb570 0x7f4c400fb940 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.834+0000 7f4c4680e640 1 -- 172.21.15.86:0/83804736 >> 172.21.15.86:0/83804736 conn(0x7f4c40072110 msgr2=0x7f4c40072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.834+0000 7f4c4680e640 1 -- 172.21.15.86:0/83804736 shutdown_connections 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.834+0000 7f4c4680e640 1 -- 172.21.15.86:0/83804736 wait complete. 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.835+0000 7f4c4680e640 1 Processor -- start 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.835+0000 7f4c4680e640 1 -- start start 2024-06-06T08:27:19.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.835+0000 7f4c4680e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c400fb570 0x7f4c40115210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.835+0000 7f4c4680e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fbe80 0x7f4c40115750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.835+0000 7f4c4680e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c40115d50 con 0x7f4c400fbe80 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.835+0000 7f4c4680e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4010f190 con 0x7f4c400fb570 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.835+0000 7f4c3f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fbe80 0x7f4c40115750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.835+0000 7f4c3f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fbe80 0x7f4c40115750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35990/0 (socket says 172.21.15.86:35990) 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.835+0000 7f4c3f7fe640 1 -- 172.21.15.86:0/3798303912 learned_addr learned my addr 172.21.15.86:0/3798303912 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c3ffff640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c400fb570 0x7f4c40115210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c3f7fe640 1 -- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c400fb570 msgr2=0x7f4c40115210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c3f7fe640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c400fb570 0x7f4c40115210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c3f7fe640 1 -- 172.21.15.86:0/3798303912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c30005040 con 0x7f4c400fbe80 2024-06-06T08:27:19.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c3f7fe640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fbe80 0x7f4c40115750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4c34002a50 tx=0x7f4c34004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:19.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c3cff9640 1 -- 172.21.15.86:0/3798303912 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c34007c30 con 0x7f4c400fbe80 2024-06-06T08:27:19.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c4680e640 1 -- 172.21.15.86:0/3798303912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4010f470 con 0x7f4c400fbe80 2024-06-06T08:27:19.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c3cff9640 1 -- 172.21.15.86:0/3798303912 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c3400f040 con 0x7f4c400fbe80 2024-06-06T08:27:19.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c3cff9640 1 -- 172.21.15.86:0/3798303912 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c3400db50 con 0x7f4c400fbe80 2024-06-06T08:27:19.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.836+0000 7f4c4680e640 1 -- 172.21.15.86:0/3798303912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4010fa10 con 0x7f4c400fbe80 2024-06-06T08:27:19.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.837+0000 7f4c267fc640 1 -- 172.21.15.86:0/3798303912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c20000fd0 con 0x7f4c400fbe80 2024-06-06T08:27:19.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.838+0000 7f4c3cff9640 1 -- 172.21.15.86:0/3798303912 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4c3401c400 con 0x7f4c400fbe80 2024-06-06T08:27:19.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.838+0000 7f4c3cff9640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4c28065430 0x7f4c280678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:19.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.838+0000 7f4c3cff9640 1 -- 172.21.15.86:0/3798303912 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4c34081d90 con 0x7f4c400fbe80 2024-06-06T08:27:19.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.838+0000 7f4c3ffff640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4c28065430 0x7f4c280678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:19.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.839+0000 7f4c3cff9640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] conn(0x7f4c2806b1b0 0x7f4c2806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:19.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.839+0000 7f4c3cff9640 1 -- 172.21.15.86:0/3798303912 --> [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c2806dc70 con 0x7f4c2806b1b0 2024-06-06T08:27:19.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.839+0000 7f4c3cff9640 1 -- 172.21.15.86:0/3798303912 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c34084340 con 0x7f4c400fbe80 2024-06-06T08:27:19.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.839+0000 7f4c44dab640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] conn(0x7f4c2806b1b0 0x7f4c2806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:19.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.839+0000 7f4c3ffff640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4c28065430 0x7f4c280678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c300047f0 tx=0x7f4c30009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:19.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.839+0000 7f4c44dab640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] conn(0x7f4c2806b1b0 0x7f4c2806d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:19.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.840+0000 7f4c3cff9640 1 -- 172.21.15.86:0/3798303912 <== osd.4 v2:172.21.15.154:6808/3195305408 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4c2806dc70 con 0x7f4c2806b1b0 2024-06-06T08:27:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.861+0000 7f4c267fc640 1 -- 172.21.15.86:0/3798303912 --> [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c20002ca0 con 0x7f4c2806b1b0 2024-06-06T08:27:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c3cff9640 1 -- 172.21.15.86:0/3798303912 <== osd.4 v2:172.21.15.154:6808/3195305408 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4c20002ca0 con 0x7f4c2806b1b0 2024-06-06T08:27:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 -- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] conn(0x7f4c2806b1b0 msgr2=0x7f4c2806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] conn(0x7f4c2806b1b0 0x7f4c2806d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 -- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4c28065430 msgr2=0x7f4c280678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4c28065430 0x7f4c280678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c300047f0 tx=0x7f4c30009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 -- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fbe80 msgr2=0x7f4c40115750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fbe80 0x7f4c40115750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4c34002a50 tx=0x7f4c34004600 comp rx=0 tx=0).stop 2024-06-06T08:27:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 -- 172.21.15.86:0/3798303912 shutdown_connections 2024-06-06T08:27:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:6808/3195305408,v1:172.21.15.154:6809/3195305408] conn(0x7f4c2806b1b0 0x7f4c2806d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4c28065430 0x7f4c280678e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c400fbe80 0x7f4c40115750 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 --2- 172.21.15.86:0/3798303912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c400fb570 0x7f4c40115210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.862+0000 7f4c267fc640 1 -- 172.21.15.86:0/3798303912 >> 172.21.15.86:0/3798303912 conn(0x7f4c40072110 msgr2=0x7f4c400fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:19.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.863+0000 7f4c267fc640 1 -- 172.21.15.86:0/3798303912 shutdown_connections 2024-06-06T08:27:19.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:19.863+0000 7f4c267fc640 1 -- 172.21.15.86:0/3798303912 wait complete. 2024-06-06T08:27:20.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:20 smithi086 ceph-mon[29114]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:20.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:20 smithi154 ceph-mon[35428]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:20.731 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:20.731 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:21.267 INFO:teuthology.orchestra.run.smithi086.stdout:146028888079 2024-06-06T08:27:21.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-06T08:27:22.300 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:22 smithi086 ceph-mon[29114]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:22.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:22 smithi154 ceph-mon[35428]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:23.315 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:23.315 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:23.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.907+0000 7f836d628640 1 -- 172.21.15.86:0/643389870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83681022c0 msgr2=0x7f8368102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:23.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.907+0000 7f836d628640 1 --2- 172.21.15.86:0/643389870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83681022c0 0x7f8368102690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f835c0048d0 tx=0x7f835c009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.907+0000 7f836d628640 1 -- 172.21.15.86:0/643389870 shutdown_connections 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.907+0000 7f836d628640 1 --2- 172.21.15.86:0/643389870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8368102c60 0x7f83680fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.907+0000 7f836d628640 1 --2- 172.21.15.86:0/643389870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83681022c0 0x7f8368102690 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.907+0000 7f836d628640 1 -- 172.21.15.86:0/643389870 >> 172.21.15.86:0/643389870 conn(0x7f83680f7e50 msgr2=0x7f83680fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.907+0000 7f836d628640 1 -- 172.21.15.86:0/643389870 shutdown_connections 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.907+0000 7f836d628640 1 -- 172.21.15.86:0/643389870 wait complete. 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f836d628640 1 Processor -- start 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f836d628640 1 -- start start 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f836d628640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83681022c0 0x7f83681133d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f836d628640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8368102c60 0x7f8368113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f836d628640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8368113fa0 con 0x7f8368102c60 2024-06-06T08:27:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f836d628640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836810d350 con 0x7f83681022c0 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f83667fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8368102c60 0x7f8368113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f83667fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8368102c60 0x7f8368113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36008/0 (socket says 172.21.15.86:36008) 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f83667fc640 1 -- 172.21.15.86:0/992818900 learned_addr learned my addr 172.21.15.86:0/992818900 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f8366ffd640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83681022c0 0x7f83681133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f83667fc640 1 -- 172.21.15.86:0/992818900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83681022c0 msgr2=0x7f83681133d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f83667fc640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83681022c0 0x7f83681133d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f83667fc640 1 -- 172.21.15.86:0/992818900 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c005040 con 0x7f8368102c60 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f8366ffd640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83681022c0 0x7f83681133d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.908+0000 7f83667fc640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8368102c60 0x7f8368113910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8358009e20 tx=0x7f83580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.909+0000 7f834f7fe640 1 -- 172.21.15.86:0/992818900 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8358010ae0 con 0x7f8368102c60 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.909+0000 7f834f7fe640 1 -- 172.21.15.86:0/992818900 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8358010c40 con 0x7f8368102c60 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.909+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836810d5d0 con 0x7f8368102c60 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.909+0000 7f834f7fe640 1 -- 172.21.15.86:0/992818900 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83580176c0 con 0x7f8368102c60 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.909+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836810daf0 con 0x7f8368102c60 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.910+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f83680fda30 con 0x7f8368102c60 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.910+0000 7f834f7fe640 1 -- 172.21.15.86:0/992818900 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f8358016070 con 0x7f8368102c60 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.910+0000 7f834f7fe640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8350065430 0x7f83500678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.910+0000 7f834f7fe640 1 -- 172.21.15.86:0/992818900 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8358086290 con 0x7f8368102c60 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.910+0000 7f8366ffd640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8350065430 0x7f83500678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.911+0000 7f834f7fe640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] conn(0x7f835006b1b0 0x7f835006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.911+0000 7f834f7fe640 1 -- 172.21.15.86:0/992818900 --> [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f835006dc70 con 0x7f835006b1b0 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.911+0000 7f834f7fe640 1 -- 172.21.15.86:0/992818900 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83580888e0 con 0x7f8368102c60 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.911+0000 7f83677fe640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] conn(0x7f835006b1b0 0x7f835006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.911+0000 7f8366ffd640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8350065430 0x7f83500678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f835c009f80 tx=0x7f835c009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.911+0000 7f83677fe640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] conn(0x7f835006b1b0 0x7f835006d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.912+0000 7f834f7fe640 1 -- 172.21.15.86:0/992818900 <== osd.1 v2:172.21.15.86:6810/1628660505 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f835006dc70 con 0x7f835006b1b0 2024-06-06T08:27:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 --> [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f836810e030 con 0x7f835006b1b0 2024-06-06T08:27:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f834f7fe640 1 -- 172.21.15.86:0/992818900 <== osd.1 v2:172.21.15.86:6810/1628660505 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f836810e030 con 0x7f835006b1b0 2024-06-06T08:27:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] conn(0x7f835006b1b0 msgr2=0x7f835006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] conn(0x7f835006b1b0 0x7f835006d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8350065430 msgr2=0x7f83500678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8350065430 0x7f83500678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f835c009f80 tx=0x7f835c009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8368102c60 msgr2=0x7f8368113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8368102c60 0x7f8368113910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8358009e20 tx=0x7f83580091d0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 shutdown_connections 2024-06-06T08:27:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6810/1628660505,v1:172.21.15.86:6811/1628660505] conn(0x7f835006b1b0 0x7f835006d5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8350065430 0x7f83500678e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8368102c60 0x7f8368113910 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 --2- 172.21.15.86:0/992818900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83681022c0 0x7f83681133d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 >> 172.21.15.86:0/992818900 conn(0x7f83680f7e50 msgr2=0x7f83680f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 shutdown_connections 2024-06-06T08:27:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:23.925+0000 7f836d628640 1 -- 172.21.15.86:0/992818900 wait complete. 2024-06-06T08:27:24.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:24 smithi086 ceph-mon[29114]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:24.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:24 smithi154 ceph-mon[35428]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:25.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:25 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:27:25.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:25 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:27:25.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 -- 172.21.15.86:0/477956423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c101b70 msgr2=0x7f8b4c0fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:25.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 --2- 172.21.15.86:0/477956423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c101b70 0x7f8b4c0fb200 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009c50 tx=0x7f8b34004d10 comp rx=0 tx=0).stop 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 -- 172.21.15.86:0/477956423 shutdown_connections 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 --2- 172.21.15.86:0/477956423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c101b70 0x7f8b4c0fb200 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 --2- 172.21.15.86:0/477956423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c1011d0 0x7f8b4c1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 -- 172.21.15.86:0/477956423 >> 172.21.15.86:0/477956423 conn(0x7f8b4c0f6dc0 msgr2=0x7f8b4c0f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 -- 172.21.15.86:0/477956423 shutdown_connections 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 -- 172.21.15.86:0/477956423 wait complete. 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 Processor -- start 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.845+0000 7f8b51b58640 1 -- start start 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b51b58640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c1011d0 0x7f8b4c113100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b51b58640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c101b70 0x7f8b4c113640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b51b58640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c113cd0 con 0x7f8b4c1011d0 2024-06-06T08:27:25.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b51b58640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c10d0c0 con 0x7f8b4c101b70 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b4b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c1011d0 0x7f8b4c113100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b4b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c1011d0 0x7f8b4c113100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36038/0 (socket says 172.21.15.86:36038) 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b4b7fe640 1 -- 172.21.15.86:0/2567644602 learned_addr learned my addr 172.21.15.86:0/2567644602 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b4affd640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c101b70 0x7f8b4c113640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b4b7fe640 1 -- 172.21.15.86:0/2567644602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c101b70 msgr2=0x7f8b4c113640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b4b7fe640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c101b70 0x7f8b4c113640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b4b7fe640 1 -- 172.21.15.86:0/2567644602 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b340098b0 con 0x7f8b4c1011d0 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b4b7fe640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c1011d0 0x7f8b4c113100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00b790 tx=0x7f8b3c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b43fff640 1 -- 172.21.15.86:0/2567644602 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b3c00c700 con 0x7f8b4c1011d0 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4c10d3a0 con 0x7f8b4c1011d0 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b43fff640 1 -- 172.21.15.86:0/2567644602 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b3c00cd30 con 0x7f8b4c1011d0 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b43fff640 1 -- 172.21.15.86:0/2567644602 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b3c013370 con 0x7f8b4c1011d0 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.846+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4c10d890 con 0x7f8b4c1011d0 2024-06-06T08:27:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.847+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4c0fc940 con 0x7f8b4c1011d0 2024-06-06T08:27:25.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.848+0000 7f8b43fff640 1 -- 172.21.15.86:0/2567644602 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f8b3c012070 con 0x7f8b4c1011d0 2024-06-06T08:27:25.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.848+0000 7f8b43fff640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b38065430 0x7f8b380678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:25.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.848+0000 7f8b43fff640 1 -- 172.21.15.86:0/2567644602 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8b3c082040 con 0x7f8b4c1011d0 2024-06-06T08:27:25.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.848+0000 7f8b4affd640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b38065430 0x7f8b380678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:25.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.849+0000 7f8b4affd640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b38065430 0x7f8b380678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009c20 tx=0x7f8b34004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:25.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:25.852+0000 7f8b43fff640 1 -- 172.21.15.86:0/2567644602 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8b3c0504f0 con 0x7f8b4c1011d0 2024-06-06T08:27:26.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.026+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8b4c0fa9c0 con 0x7f8b4c1011d0 2024-06-06T08:27:26.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.029+0000 7f8b43fff640 1 -- 172.21.15.86:0/2567644602 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8b3c053b10 con 0x7f8b4c1011d0 2024-06-06T08:27:26.028 INFO:teuthology.orchestra.run.smithi086.stdout:167503724554 2024-06-06T08:27:26.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b38065430 msgr2=0x7f8b380678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:26.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b38065430 0x7f8b380678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009c20 tx=0x7f8b34004d10 comp rx=0 tx=0).stop 2024-06-06T08:27:26.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c1011d0 msgr2=0x7f8b4c113100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:26.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c1011d0 0x7f8b4c113100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00b790 tx=0x7f8b3c0091d0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 shutdown_connections 2024-06-06T08:27:26.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c101b70 0x7f8b4c113640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8b38065430 0x7f8b380678e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 --2- 172.21.15.86:0/2567644602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b4c1011d0 0x7f8b4c113100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 >> 172.21.15.86:0/2567644602 conn(0x7f8b4c0f6dc0 msgr2=0x7f8b4c0657d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:26.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 shutdown_connections 2024-06-06T08:27:26.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.031+0000 7f8b51b58640 1 -- 172.21.15.86:0/2567644602 wait complete. 2024-06-06T08:27:26.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:26 smithi086 ceph-mon[29114]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:26.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:26 smithi154 ceph-mon[35428]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 -- 172.21.15.86:0/2977077468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 msgr2=0x7fa2081015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 --2- 172.21.15.86:0/2977077468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 0x7fa2081015a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa200009c50 tx=0x7fa200004d10 comp rx=0 tx=0).stop 2024-06-06T08:27:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 -- 172.21.15.86:0/2977077468 shutdown_connections 2024-06-06T08:27:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 --2- 172.21.15.86:0/2977077468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208101b70 0x7fa2080fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 --2- 172.21.15.86:0/2977077468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 0x7fa2081015a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 -- 172.21.15.86:0/2977077468 >> 172.21.15.86:0/2977077468 conn(0x7fa2080f6dc0 msgr2=0x7fa2080f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 -- 172.21.15.86:0/2977077468 shutdown_connections 2024-06-06T08:27:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 -- 172.21.15.86:0/2977077468 wait complete. 2024-06-06T08:27:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 Processor -- start 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.508+0000 7fa20cb9a640 1 -- start start 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20cb9a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 0x7fa208111f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20cb9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208101b70 0x7fa2081124d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20cb9a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa208112b60 con 0x7fa2081011d0 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20cb9a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20810bf10 con 0x7fa208101b70 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 0x7fa208111f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 0x7fa208111f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36052/0 (socket says 172.21.15.86:36052) 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20659c640 1 -- 172.21.15.86:0/302575639 learned_addr learned my addr 172.21.15.86:0/302575639 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20659c640 1 -- 172.21.15.86:0/302575639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208101b70 msgr2=0x7fa2081124d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa205d9b640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208101b70 0x7fa2081124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20659c640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208101b70 0x7fa2081124d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20659c640 1 -- 172.21.15.86:0/302575639 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2000098b0 con 0x7fa2081011d0 2024-06-06T08:27:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20659c640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 0x7fa208111f90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa200004f80 tx=0x7fa200004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa1f6ffd640 1 -- 172.21.15.86:0/302575639 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa200008de0 con 0x7fa2081011d0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa1f6ffd640 1 -- 172.21.15.86:0/302575639 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa200032c00 con 0x7fa2081011d0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20810c190 con 0x7fa2081011d0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa1f6ffd640 1 -- 172.21.15.86:0/302575639 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa20003dbc0 con 0x7fa2081011d0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.509+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20810c6b0 con 0x7fa2081011d0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.511+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2080fc940 con 0x7fa2081011d0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.511+0000 7fa1f6ffd640 1 -- 172.21.15.86:0/302575639 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa20002f030 con 0x7fa2081011d0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.511+0000 7fa1f6ffd640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa1f0065430 0x7fa1f00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.511+0000 7fa1f6ffd640 1 -- 172.21.15.86:0/302575639 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa2000a3aa0 con 0x7fa2081011d0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.511+0000 7fa205d9b640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa1f0065430 0x7fa1f00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.512+0000 7fa205d9b640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa1f0065430 0x7fa1f00678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa20810d850 tx=0x7fa1fc008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:26.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.515+0000 7fa1f6ffd640 1 -- 172.21.15.86:0/302575639 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa200072080 con 0x7fa2081011d0 2024-06-06T08:27:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.685+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa2080fa9c0 con 0x7fa2081011d0 2024-06-06T08:27:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.686+0000 7fa1f6ffd640 1 -- 172.21.15.86:0/302575639 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa200032d70 con 0x7fa2081011d0 2024-06-06T08:27:26.685 INFO:teuthology.orchestra.run.smithi086.stdout:146028888080 2024-06-06T08:27:26.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.687+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa1f0065430 msgr2=0x7fa1f00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.687+0000 7fa20cb9a640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa1f0065430 0x7fa1f00678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa20810d850 tx=0x7fa1fc008040 comp rx=0 tx=0).stop 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.687+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 msgr2=0x7fa208111f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.687+0000 7fa20cb9a640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 0x7fa208111f90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa200004f80 tx=0x7fa200004d10 comp rx=0 tx=0).stop 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.688+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 shutdown_connections 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.688+0000 7fa20cb9a640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208101b70 0x7fa2081124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.688+0000 7fa20cb9a640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa1f0065430 0x7fa1f00678e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.688+0000 7fa20cb9a640 1 --2- 172.21.15.86:0/302575639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2081011d0 0x7fa208111f90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.688+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 >> 172.21.15.86:0/302575639 conn(0x7fa2080f6dc0 msgr2=0x7fa2080f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.688+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 shutdown_connections 2024-06-06T08:27:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:26.688+0000 7fa20cb9a640 1 -- 172.21.15.86:0/302575639 wait complete. 2024-06-06T08:27:27.019 INFO:teuthology.orchestra.run.smithi086.stdout:60129542174 2024-06-06T08:27:27.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-06T08:27:27.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:27 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2567644602' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-06T08:27:27.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:27 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/302575639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-06T08:27:27.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:27 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2567644602' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-06T08:27:27.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:27 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/302575639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-06T08:27:28.254 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:28 smithi086 ceph-mon[29114]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:28.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:28 smithi154 ceph-mon[35428]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:28.636 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724554 for osd.5 2024-06-06T08:27:28.636 DEBUG:teuthology.parallel:result is None 2024-06-06T08:27:29.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.176+0000 7fa6b2212640 1 -- 172.21.15.86:0/3473669810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 msgr2=0x7fa6ac106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:29.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.176+0000 7fa6b2212640 1 --2- 172.21.15.86:0/3473669810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 0x7fa6ac106330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0004900 tx=0x7fa6a0009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.176+0000 7fa6b2212640 1 -- 172.21.15.86:0/3473669810 shutdown_connections 2024-06-06T08:27:29.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.176+0000 7fa6b2212640 1 --2- 172.21.15.86:0/3473669810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 0x7fa6ac106330 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.176+0000 7fa6b2212640 1 --2- 172.21.15.86:0/3473669810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6ac0fd2c0 0x7fa6ac0fd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.176+0000 7fa6b2212640 1 -- 172.21.15.86:0/3473669810 >> 172.21.15.86:0/3473669810 conn(0x7fa6ac0f6d60 msgr2=0x7fa6ac0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:29.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.176+0000 7fa6b2212640 1 -- 172.21.15.86:0/3473669810 shutdown_connections 2024-06-06T08:27:29.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.176+0000 7fa6b2212640 1 -- 172.21.15.86:0/3473669810 wait complete. 2024-06-06T08:27:29.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.177+0000 7fa6b2212640 1 Processor -- start 2024-06-06T08:27:29.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.177+0000 7fa6b2212640 1 -- start start 2024-06-06T08:27:29.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.177+0000 7fa6b2212640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6ac0fd2c0 0x7fa6ac113140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.177+0000 7fa6b2212640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 0x7fa6ac113680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.177+0000 7fa6b2212640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac113d10 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.177+0000 7fa6b2212640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac10d0c0 con 0x7fa6ac0fd2c0 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.177+0000 7fa6aaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 0x7fa6ac113680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.177+0000 7fa6aaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 0x7fa6ac113680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57272/0 (socket says 172.21.15.86:57272) 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.177+0000 7fa6aaffd640 1 -- 172.21.15.86:0/4212908864 learned_addr learned my addr 172.21.15.86:0/4212908864 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa6ab7fe640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6ac0fd2c0 0x7fa6ac113140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa6aaffd640 1 -- 172.21.15.86:0/4212908864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6ac0fd2c0 msgr2=0x7fa6ac113140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa6aaffd640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6ac0fd2c0 0x7fa6ac113140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa6aaffd640 1 -- 172.21.15.86:0/4212908864 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a0005040 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa6ab7fe640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6ac0fd2c0 0x7fa6ac113140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa6aaffd640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 0x7fa6ac113680 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0003080 tx=0x7fa6a0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa693fff640 1 -- 172.21.15.86:0/4212908864 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6a0038070 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac10d340 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa693fff640 1 -- 172.21.15.86:0/4212908864 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6a00338f0 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa693fff640 1 -- 172.21.15.86:0/4212908864 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6a003c810 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.178+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac10d7a0 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.179+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac066010 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.179+0000 7fa693fff640 1 -- 172.21.15.86:0/4212908864 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa6a0030020 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.180+0000 7fa693fff640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa694065430 0x7fa6940678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.180+0000 7fa693fff640 1 -- 172.21.15.86:0/4212908864 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa6a00a2270 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.180+0000 7fa6ab7fe640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa694065430 0x7fa6940678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:29.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.180+0000 7fa6ab7fe640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa694065430 0x7fa6940678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa69c006fd0 tx=0x7fa69c008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:29.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.184+0000 7fa693fff640 1 -- 172.21.15.86:0/4212908864 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa6a0070720 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.353+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa6ac0fa9a0 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.353+0000 7fa693fff640 1 -- 172.21.15.86:0/4212908864 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa6a0073d40 con 0x7fa6ac0fdc60 2024-06-06T08:27:29.352 INFO:teuthology.orchestra.run.smithi086.stdout:81604378651 2024-06-06T08:27:29.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa694065430 msgr2=0x7fa6940678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:29.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa694065430 0x7fa6940678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa69c006fd0 tx=0x7fa69c008040 comp rx=0 tx=0).stop 2024-06-06T08:27:29.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 msgr2=0x7fa6ac113680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:29.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 0x7fa6ac113680 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0003080 tx=0x7fa6a0009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 shutdown_connections 2024-06-06T08:27:29.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa694065430 0x7fa6940678e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0fdc60 0x7fa6ac113680 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 --2- 172.21.15.86:0/4212908864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6ac0fd2c0 0x7fa6ac113140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 >> 172.21.15.86:0/4212908864 conn(0x7fa6ac0f6d60 msgr2=0x7fa6ac0f8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:29.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 shutdown_connections 2024-06-06T08:27:29.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.355+0000 7fa6b2212640 1 -- 172.21.15.86:0/4212908864 wait complete. 2024-06-06T08:27:29.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.780+0000 7f4e366f1640 1 -- 172.21.15.86:0/425794301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fbe80 msgr2=0x7f4e300fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:29.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.780+0000 7f4e366f1640 1 --2- 172.21.15.86:0/425794301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fbe80 0x7f4e300fcd80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4e240048d0 tx=0x7f4e24009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 -- 172.21.15.86:0/425794301 shutdown_connections 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 --2- 172.21.15.86:0/425794301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fbe80 0x7f4e300fcd80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 --2- 172.21.15.86:0/425794301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300fb570 0x7f4e300fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 -- 172.21.15.86:0/425794301 >> 172.21.15.86:0/425794301 conn(0x7f4e30072110 msgr2=0x7f4e30072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 -- 172.21.15.86:0/425794301 shutdown_connections 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 -- 172.21.15.86:0/425794301 wait complete. 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 Processor -- start 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 -- start start 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fb570 0x7f4e3010bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300fbe80 0x7f4e3010c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3010c800 con 0x7f4e300fb570 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.781+0000 7f4e366f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3010c940 con 0x7f4e300fbe80 2024-06-06T08:27:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fb570 0x7f4e3010bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fb570 0x7f4e3010bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57282/0 (socket says 172.21.15.86:57282) 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2ffff640 1 -- 172.21.15.86:0/3188376094 learned_addr learned my addr 172.21.15.86:0/3188376094 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2f7fe640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300fbe80 0x7f4e3010c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2ffff640 1 -- 172.21.15.86:0/3188376094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300fbe80 msgr2=0x7f4e3010c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2ffff640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300fbe80 0x7f4e3010c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2ffff640 1 -- 172.21.15.86:0/3188376094 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24005040 con 0x7f4e300fb570 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2f7fe640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300fbe80 0x7f4e3010c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2ffff640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fb570 0x7f4e3010bd80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e20002a50 tx=0x7f4e20004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2cff9640 1 -- 172.21.15.86:0/3188376094 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e20007c30 con 0x7f4e300fb570 2024-06-06T08:27:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e301193f0 con 0x7f4e300fb570 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2cff9640 1 -- 172.21.15.86:0/3188376094 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e2000f040 con 0x7f4e300fb570 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e2cff9640 1 -- 172.21.15.86:0/3188376094 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e2000db50 con 0x7f4e300fb570 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.782+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e301197a0 con 0x7f4e300fb570 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.783+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e300fc920 con 0x7f4e300fb570 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.783+0000 7f4e2cff9640 1 -- 172.21.15.86:0/3188376094 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4e2001c400 con 0x7f4e300fb570 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.784+0000 7f4e2cff9640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4e18065430 0x7f4e180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.784+0000 7f4e2cff9640 1 -- 172.21.15.86:0/3188376094 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4e20081d70 con 0x7f4e300fb570 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.784+0000 7f4e2f7fe640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4e18065430 0x7f4e180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.784+0000 7f4e2f7fe640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4e18065430 0x7f4e180678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e30065790 tx=0x7f4e24035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.788+0000 7f4e2cff9640 1 -- 172.21.15.86:0/3188376094 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4e2004cdf0 con 0x7f4e300fb570 2024-06-06T08:27:29.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.957+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4e30072c20 con 0x7f4e300fb570 2024-06-06T08:27:29.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.958+0000 7f4e2cff9640 1 -- 172.21.15.86:0/3188376094 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4e200502e0 con 0x7f4e300fb570 2024-06-06T08:27:29.957 INFO:teuthology.orchestra.run.smithi086.stdout:111669149717 2024-06-06T08:27:29.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4e18065430 msgr2=0x7f4e180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:29.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4e18065430 0x7f4e180678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e30065790 tx=0x7f4e24035040 comp rx=0 tx=0).stop 2024-06-06T08:27:29.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fb570 msgr2=0x7f4e3010bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:29.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fb570 0x7f4e3010bd80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e20002a50 tx=0x7f4e20004600 comp rx=0 tx=0).stop 2024-06-06T08:27:29.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 shutdown_connections 2024-06-06T08:27:29.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300fbe80 0x7f4e3010c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4e18065430 0x7f4e180678e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 --2- 172.21.15.86:0/3188376094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e300fb570 0x7f4e3010bd80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:29.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 >> 172.21.15.86:0/3188376094 conn(0x7f4e30072110 msgr2=0x7f4e300ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:29.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 shutdown_connections 2024-06-06T08:27:29.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:29.960+0000 7f4e366f1640 1 -- 172.21.15.86:0/3188376094 wait complete. 2024-06-06T08:27:29.979 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888080 for osd.4 2024-06-06T08:27:29.980 DEBUG:teuthology.parallel:result is None 2024-06-06T08:27:30.338 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:30 smithi086 ceph-mon[29114]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:30.338 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:30 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/4212908864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-06T08:27:30.338 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:30 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3188376094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-06T08:27:30.375 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:30 smithi154 ceph-mon[35428]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:30 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/4212908864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-06T08:27:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:30 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3188376094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-06T08:27:30.755 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378651 for osd.2 2024-06-06T08:27:30.755 DEBUG:teuthology.parallel:result is None 2024-06-06T08:27:31.325 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149715 got 111669149717 for osd.3 2024-06-06T08:27:31.325 DEBUG:teuthology.parallel:result is None 2024-06-06T08:27:31.366 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:32.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:32 smithi086 ceph-mon[29114]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:32.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:32 smithi154 ceph-mon[35428]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.202+0000 7f75a478a640 1 -- 172.21.15.86:0/2745847104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c1011d0 msgr2=0x7f759c1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.202+0000 7f75a478a640 1 --2- 172.21.15.86:0/2745847104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c1011d0 0x7f759c1015a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f75980048d0 tx=0x7f7598009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.202+0000 7f75a478a640 1 -- 172.21.15.86:0/2745847104 shutdown_connections 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.202+0000 7f75a478a640 1 --2- 172.21.15.86:0/2745847104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f759c101b70 0x7f759c0fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.202+0000 7f75a478a640 1 --2- 172.21.15.86:0/2745847104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c1011d0 0x7f759c1015a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.202+0000 7f75a478a640 1 -- 172.21.15.86:0/2745847104 >> 172.21.15.86:0/2745847104 conn(0x7f759c0f6dc0 msgr2=0x7f759c0f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.202+0000 7f75a478a640 1 -- 172.21.15.86:0/2745847104 shutdown_connections 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.202+0000 7f75a478a640 1 -- 172.21.15.86:0/2745847104 wait complete. 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.202+0000 7f75a478a640 1 Processor -- start 2024-06-06T08:27:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a478a640 1 -- start start 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a478a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f759c1011d0 0x7f759c113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a478a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c101b70 0x7f759c113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a478a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c113b40 con 0x7f759c101b70 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a478a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c10cf80 con 0x7f759c1011d0 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a1d25640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c101b70 0x7f759c113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a1d25640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c101b70 0x7f759c113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57292/0 (socket says 172.21.15.86:57292) 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a1d25640 1 -- 172.21.15.86:0/392102557 learned_addr learned my addr 172.21.15.86:0/392102557 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a2526640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f759c1011d0 0x7f759c113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a1d25640 1 -- 172.21.15.86:0/392102557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f759c1011d0 msgr2=0x7f759c113000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a1d25640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f759c1011d0 0x7f759c113000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a1d25640 1 -- 172.21.15.86:0/392102557 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7598005040 con 0x7f759c101b70 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f75a1d25640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c101b70 0x7f759c113540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f759000c830 tx=0x7f7590004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.203+0000 7f758effd640 1 -- 172.21.15.86:0/392102557 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f759000a7d0 con 0x7f759c101b70 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.204+0000 7f758effd640 1 -- 172.21.15.86:0/392102557 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f759000ae00 con 0x7f759c101b70 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.204+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c10d260 con 0x7f759c101b70 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.204+0000 7f758effd640 1 -- 172.21.15.86:0/392102557 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75900113a0 con 0x7f759c101b70 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.204+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c10d800 con 0x7f759c101b70 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.204+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759c0fc940 con 0x7f759c101b70 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.205+0000 7f758effd640 1 -- 172.21.15.86:0/392102557 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f7590012400 con 0x7f759c101b70 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.205+0000 7f758effd640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7588065430 0x7f75880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.205+0000 7f758effd640 1 -- 172.21.15.86:0/392102557 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f75900825b0 con 0x7f759c101b70 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.205+0000 7f75a2526640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7588065430 0x7f75880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.206+0000 7f75a2526640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7588065430 0x7f75880678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7598009f80 tx=0x7f7598035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.210+0000 7f758effd640 1 -- 172.21.15.86:0/392102557 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7590050ae0 con 0x7f759c101b70 2024-06-06T08:27:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.377+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f759c0fa9c0 con 0x7f759c101b70 2024-06-06T08:27:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.377+0000 7f758effd640 1 -- 172.21.15.86:0/392102557 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7590054100 con 0x7f759c101b70 2024-06-06T08:27:33.376 INFO:teuthology.orchestra.run.smithi086.stdout:38654705702 2024-06-06T08:27:33.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7588065430 msgr2=0x7f75880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:33.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7588065430 0x7f75880678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7598009f80 tx=0x7f7598035040 comp rx=0 tx=0).stop 2024-06-06T08:27:33.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c101b70 msgr2=0x7f759c113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:33.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c101b70 0x7f759c113540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f759000c830 tx=0x7f7590004430 comp rx=0 tx=0).stop 2024-06-06T08:27:33.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 shutdown_connections 2024-06-06T08:27:33.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7588065430 0x7f75880678e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:33.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f759c101b70 0x7f759c113540 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:33.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 --2- 172.21.15.86:0/392102557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f759c1011d0 0x7f759c113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:33.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 >> 172.21.15.86:0/392102557 conn(0x7f759c0f6dc0 msgr2=0x7f759c065800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 shutdown_connections 2024-06-06T08:27:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:33.379+0000 7f75a478a640 1 -- 172.21.15.86:0/392102557 wait complete. 2024-06-06T08:27:34.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:34 smithi086 ceph-mon[29114]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:34.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:34 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/392102557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-06T08:27:34.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.308+0000 7fe8b230e640 1 -- 172.21.15.86:0/3398240342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 msgr2=0x7fe8ac102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:34.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.308+0000 7fe8b230e640 1 --2- 172.21.15.86:0/3398240342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 0x7fe8ac102670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0048d0 tx=0x7fe89c009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:34.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.308+0000 7fe8b230e640 1 -- 172.21.15.86:0/3398240342 shutdown_connections 2024-06-06T08:27:34.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.308+0000 7fe8b230e640 1 --2- 172.21.15.86:0/3398240342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8ac102c40 0x7fe8ac0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:34.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.308+0000 7fe8b230e640 1 --2- 172.21.15.86:0/3398240342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 0x7fe8ac102670 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:34.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.308+0000 7fe8b230e640 1 -- 172.21.15.86:0/3398240342 >> 172.21.15.86:0/3398240342 conn(0x7fe8ac0f7e50 msgr2=0x7fe8ac0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:34.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.308+0000 7fe8b230e640 1 -- 172.21.15.86:0/3398240342 shutdown_connections 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.308+0000 7fe8b230e640 1 -- 172.21.15.86:0/3398240342 wait complete. 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.309+0000 7fe8b230e640 1 Processor -- start 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.309+0000 7fe8b230e640 1 -- start start 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.309+0000 7fe8b230e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 0x7fe8ac113060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.309+0000 7fe8b230e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8ac102c40 0x7fe8ac1135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.309+0000 7fe8b230e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac113c30 con 0x7fe8ac1022a0 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.309+0000 7fe8b230e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac10cfe0 con 0x7fe8ac102c40 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.309+0000 7fe8abfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 0x7fe8ac113060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.309+0000 7fe8abfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 0x7fe8ac113060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57306/0 (socket says 172.21.15.86:57306) 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.309+0000 7fe8abfff640 1 -- 172.21.15.86:0/1053659892 learned_addr learned my addr 172.21.15.86:0/1053659892 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8ab7fe640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8ac102c40 0x7fe8ac1135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8abfff640 1 -- 172.21.15.86:0/1053659892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8ac102c40 msgr2=0x7fe8ac1135a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8abfff640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8ac102c40 0x7fe8ac1135a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8abfff640 1 -- 172.21.15.86:0/1053659892 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c005040 con 0x7fe8ac1022a0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8ab7fe640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8ac102c40 0x7fe8ac1135a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8abfff640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 0x7fe8ac113060 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe89c003080 tx=0x7fe89c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8a8ff9640 1 -- 172.21.15.86:0/1053659892 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89c038070 con 0x7fe8ac1022a0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ac10d260 con 0x7fe8ac1022a0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8a8ff9640 1 -- 172.21.15.86:0/1053659892 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe89c008ee0 con 0x7fe8ac1022a0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8a8ff9640 1 -- 172.21.15.86:0/1053659892 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89c03d870 con 0x7fe8ac1022a0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.310+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ac10d720 con 0x7fe8ac1022a0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.311+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ac0fda10 con 0x7fe8ac1022a0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.311+0000 7fe8a8ff9640 1 -- 172.21.15.86:0/1053659892 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe89c030020 con 0x7fe8ac1022a0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.312+0000 7fe8a8ff9640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe894065430 0x7fe8940678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.312+0000 7fe8ab7fe640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe894065430 0x7fe8940678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.312+0000 7fe8a8ff9640 1 -- 172.21.15.86:0/1053659892 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe89c0a2220 con 0x7fe8ac1022a0 2024-06-06T08:27:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.312+0000 7fe8ab7fe640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe894065430 0x7fe8940678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac10e920 tx=0x7fe8a0008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:34.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.316+0000 7fe8a8ff9640 1 -- 172.21.15.86:0/1053659892 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe89c070750 con 0x7fe8ac1022a0 2024-06-06T08:27:34.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:34 smithi154 ceph-mon[35428]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:34.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:34 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/392102557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-06T08:27:34.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.486+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe8ac0fba90 con 0x7fe8ac1022a0 2024-06-06T08:27:34.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.486+0000 7fe8a8ff9640 1 -- 172.21.15.86:0/1053659892 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe89c073d70 con 0x7fe8ac1022a0 2024-06-06T08:27:34.485 INFO:teuthology.orchestra.run.smithi086.stdout:60129542176 2024-06-06T08:27:34.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe894065430 msgr2=0x7fe8940678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:34.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe894065430 0x7fe8940678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac10e920 tx=0x7fe8a0008040 comp rx=0 tx=0).stop 2024-06-06T08:27:34.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 msgr2=0x7fe8ac113060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:34.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 0x7fe8ac113060 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe89c003080 tx=0x7fe89c009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:34.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 shutdown_connections 2024-06-06T08:27:34.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8ac102c40 0x7fe8ac1135a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:34.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe894065430 0x7fe8940678e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:34.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 --2- 172.21.15.86:0/1053659892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ac1022a0 0x7fe8ac113060 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:34.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 >> 172.21.15.86:0/1053659892 conn(0x7fe8ac0f7e50 msgr2=0x7fe8ac0f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:34.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 shutdown_connections 2024-06-06T08:27:34.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:34.488+0000 7fe8b230e640 1 -- 172.21.15.86:0/1053659892 wait complete. 2024-06-06T08:27:34.731 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705702 for osd.0 2024-06-06T08:27:34.731 DEBUG:teuthology.parallel:result is None 2024-06-06T08:27:35.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:35 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1053659892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-06T08:27:35.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:35 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1053659892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-06T08:27:35.454 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-06-06T08:27:35.454 DEBUG:teuthology.parallel:result is None 2024-06-06T08:27:35.458 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-06T08:27:35.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-06T08:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:36.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:36 smithi086 ceph-mon[29114]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:36.318 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:36 smithi154 ceph-mon[35428]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:37.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.124+0000 7f031ac1c640 1 -- 172.21.15.86:0/1240976386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 msgr2=0x7f0314102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:37.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.124+0000 7f031ac1c640 1 --2- 172.21.15.86:0/1240976386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 0x7f0314102690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f03080048d0 tx=0x7f0308009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:37.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.124+0000 7f031ac1c640 1 -- 172.21.15.86:0/1240976386 shutdown_connections 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.124+0000 7f031ac1c640 1 --2- 172.21.15.86:0/1240976386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0314102c60 0x7f03140fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.124+0000 7f031ac1c640 1 --2- 172.21.15.86:0/1240976386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 0x7f0314102690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.124+0000 7f031ac1c640 1 -- 172.21.15.86:0/1240976386 >> 172.21.15.86:0/1240976386 conn(0x7f03140f7e50 msgr2=0x7f03140fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.124+0000 7f031ac1c640 1 -- 172.21.15.86:0/1240976386 shutdown_connections 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.124+0000 7f031ac1c640 1 -- 172.21.15.86:0/1240976386 wait complete. 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.125+0000 7f031ac1c640 1 Processor -- start 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.125+0000 7f031ac1c640 1 -- start start 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.125+0000 7f031ac1c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 0x7f031406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.125+0000 7f031ac1c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0314102c60 0x7f031406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.125+0000 7f031ac1c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03140701f0 con 0x7f03141022c0 2024-06-06T08:27:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.125+0000 7f031ac1c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0314070360 con 0x7f0314102c60 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.125+0000 7f03189b8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 0x7f031406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.125+0000 7f03189b8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 0x7f031406f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51800/0 (socket says 172.21.15.86:51800) 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.125+0000 7f03189b8640 1 -- 172.21.15.86:0/1483311346 learned_addr learned my addr 172.21.15.86:0/1483311346 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f0313fff640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0314102c60 0x7f031406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f03189b8640 1 -- 172.21.15.86:0/1483311346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0314102c60 msgr2=0x7f031406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f03189b8640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0314102c60 0x7f031406fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f03189b8640 1 -- 172.21.15.86:0/1483311346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0308005040 con 0x7f03141022c0 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f0313fff640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0314102c60 0x7f031406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f03189b8640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 0x7f031406f770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f03080043d0 tx=0x7f0308009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:37.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f03117fa640 1 -- 172.21.15.86:0/1483311346 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0308038070 con 0x7f03141022c0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f03117fa640 1 -- 172.21.15.86:0/1483311346 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0308032970 con 0x7f03141022c0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0314070590 con 0x7f03141022c0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f03117fa640 1 -- 172.21.15.86:0/1483311346 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f030803da60 con 0x7f03141022c0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.126+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03141199b0 con 0x7f03141022c0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.127+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03140fda30 con 0x7f03141022c0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.127+0000 7f03117fa640 1 -- 172.21.15.86:0/1483311346 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0308030020 con 0x7f03141022c0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.128+0000 7f03117fa640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0304065430 0x7f03040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.128+0000 7f0313fff640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0304065430 0x7f03040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.128+0000 7f03117fa640 1 -- 172.21.15.86:0/1483311346 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f03080a26e0 con 0x7f03141022c0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.128+0000 7f0313fff640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0304065430 0x7f03040678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0300007fd0 tx=0x7f0300009e90 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:37.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.132+0000 7f03117fa640 1 -- 172.21.15.86:0/1483311346 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0308070b90 con 0x7f03141022c0 2024-06-06T08:27:37.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.295+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 --> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0314070e60 con 0x7f0304065430 2024-06-06T08:27:37.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.296+0000 7f03117fa640 1 -- 172.21.15.86:0/1483311346 <== mgr.14212 v2:172.21.15.86:6800/3471926911 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27858 (secure 0 0 0) 0x7f0314070e60 con 0x7f0304065430 2024-06-06T08:27:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:27:37.299 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-06T08:27:37.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.298+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0304065430 msgr2=0x7f03040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:37.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.298+0000 7f031ac1c640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0304065430 0x7f03040678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0300007fd0 tx=0x7f0300009e90 comp rx=0 tx=0).stop 2024-06-06T08:27:37.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.298+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 msgr2=0x7f031406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:37.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.298+0000 7f031ac1c640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 0x7f031406f770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f03080043d0 tx=0x7f0308009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:37.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.298+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 shutdown_connections 2024-06-06T08:27:37.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.298+0000 7f031ac1c640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0314102c60 0x7f031406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:37.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.298+0000 7f031ac1c640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0304065430 0x7f03040678e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:37.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.298+0000 7f031ac1c640 1 --2- 172.21.15.86:0/1483311346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03141022c0 0x7f031406f770 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:37.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.298+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 >> 172.21.15.86:0/1483311346 conn(0x7f03140f7e50 msgr2=0x7f03140f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:37.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.299+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 shutdown_connections 2024-06-06T08:27:37.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:37.299+0000 7f031ac1c640 1 -- 172.21.15.86:0/1483311346 wait complete. 2024-06-06T08:27:37.900 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-06-06T08:27:36.691390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788036,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682876,"statfs":{"total":575970213888,"available":574139265024,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002125"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-06T08:26:45.458910+0000","last_change":"2024-06-06T08:26:20.024006+0000","last_active":"2024-06-06T08:26:45.458910+0000","last_peered":"2024-06-06T08:26:45.458910+0000","last_clean":"2024-06-06T08:26:45.458910+0000","last_became_active":"2024-06-06T08:26:20.023753+0000","last_became_peered":"2024-06-06T08:26:20.023753+0000","last_unstale":"2024-06-06T08:26:45.458910+0000","last_undegraded":"2024-06-06T08:26:45.458910+0000","last_fullsized":"2024-06-06T08:26:45.458910+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-06T08:25:32.961935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-06T08:25:32.961935+0000","last_clean_scrub_stamp":"2024-06-06T08:25:32.961935+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-07T13:13:12.879348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 08:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jun 6 08:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Jun 6 08:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.378}]},{"osd":3,"last update":"Thu Jun 6 08:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.25800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 08:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jun 6 08:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.37}]},{"osd":2,"last update":"Thu Jun 6 08:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.441},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.432}]},{"osd":4,"last update":"Thu Jun 6 08:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.315}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 08:26:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"last":0.23699999999999999}]},{"osd":1,"last update":"Thu Jun 6 08:26:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.248},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jun 6 08:26:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.434}]},{"osd":4,"last update":"Thu Jun 6 08:27:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.374},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.44700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 08:27:08 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.33000000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.33900000000000002,"15min":0.33600000000000002},"min":{"1min":0.26600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.495,"5min":0.499,"15min":0.499},"last":0.251}]},{"osd":2,"last update":"Thu Jun 6 08:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.46100000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.379,"15min":0.38700000000000001},"min":{"1min":0.255,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.47199999999999998,"5min":0.495,"15min":0.495},"last":0.27500000000000002}]},{"osd":3,"last update":"Thu Jun 6 08:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Thu Jun 6 08:27:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 6 08:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.313,"15min":0.311},"min":{"1min":0.22900000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.45700000000000002,"5min":0.496,"15min":0.496},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.317},"min":{"1min":0.24199999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.375}]},{"osd":2,"last update":"Thu Jun 6 08:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.34100000000000003}]},{"osd":3,"last update":"Thu Jun 6 08:26:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.754},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jun 6 08:27:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-06T08:27:37.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-06T08:27:38.153 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:38.276 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:38 smithi086 ceph-mon[29114]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:38.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:38 smithi154 ceph-mon[35428]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:39.249 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:39 smithi086 ceph-mon[29114]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T08:27:39.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:39 smithi154 ceph-mon[35428]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T08:27:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.702+0000 7f861cb78640 1 -- 172.21.15.86:0/4038791335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86181011d0 msgr2=0x7f86181015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.702+0000 7f861cb78640 1 --2- 172.21.15.86:0/4038791335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86181011d0 0x7f86181015a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f86080048d0 tx=0x7f8608009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.702+0000 7f861cb78640 1 -- 172.21.15.86:0/4038791335 shutdown_connections 2024-06-06T08:27:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.702+0000 7f861cb78640 1 --2- 172.21.15.86:0/4038791335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8618101b70 0x7f86180fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.702+0000 7f861cb78640 1 --2- 172.21.15.86:0/4038791335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86181011d0 0x7f86181015a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.702+0000 7f861cb78640 1 -- 172.21.15.86:0/4038791335 >> 172.21.15.86:0/4038791335 conn(0x7f86180f6dc0 msgr2=0x7f86180f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.702+0000 7f861cb78640 1 -- 172.21.15.86:0/4038791335 shutdown_connections 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.702+0000 7f861cb78640 1 -- 172.21.15.86:0/4038791335 wait complete. 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f861cb78640 1 Processor -- start 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f861cb78640 1 -- start start 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f861cb78640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86181011d0 0x7f861810dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f861cb78640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8618101b70 0x7f861810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f861cb78640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861810eae0 con 0x7f8618101b70 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f861cb78640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861810ec50 con 0x7f86181011d0 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f860ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8618101b70 0x7f861810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f860ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8618101b70 0x7f861810e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51828/0 (socket says 172.21.15.86:51828) 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f860ffff640 1 -- 172.21.15.86:0/4067161191 learned_addr learned my addr 172.21.15.86:0/4067161191 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f860ffff640 1 -- 172.21.15.86:0/4067161191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86181011d0 msgr2=0x7f861810dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f860ffff640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86181011d0 0x7f861810dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.703+0000 7f860ffff640 1 -- 172.21.15.86:0/4067161191 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8608005040 con 0x7f8618101b70 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.704+0000 7f860ffff640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8618101b70 0x7f861810e4e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f860000c830 tx=0x7f8600004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.704+0000 7f860effd640 1 -- 172.21.15.86:0/4067161191 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f860000a850 con 0x7f8618101b70 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.704+0000 7f860effd640 1 -- 172.21.15.86:0/4067161191 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f860000ae80 con 0x7f8618101b70 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.704+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8618117010 con 0x7f8618101b70 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.704+0000 7f860effd640 1 -- 172.21.15.86:0/4067161191 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8600012ea0 con 0x7f8618101b70 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.704+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8618117550 con 0x7f8618101b70 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.705+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86180fc940 con 0x7f8618101b70 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.705+0000 7f860effd640 1 -- 172.21.15.86:0/4067161191 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f8600012400 con 0x7f8618101b70 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.705+0000 7f860effd640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8604065430 0x7f86040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.706+0000 7f860effd640 1 -- 172.21.15.86:0/4067161191 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8600082620 con 0x7f8618101b70 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.706+0000 7f861659c640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8604065430 0x7f86040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.706+0000 7f861659c640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8604065430 0x7f86040678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8608032430 tx=0x7f8608009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:39.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.710+0000 7f860effd640 1 -- 172.21.15.86:0/4067161191 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8600050cb0 con 0x7f8618101b70 2024-06-06T08:27:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.872+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 --> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f861804b480 con 0x7f8604065430 2024-06-06T08:27:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.874+0000 7f860effd640 1 -- 172.21.15.86:0/4067161191 <== mgr.14212 v2:172.21.15.86:6800/3471926911 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27914 (secure 0 0 0) 0x7f861804b480 con 0x7f8604065430 2024-06-06T08:27:39.873 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:27:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-06T08:27:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8604065430 msgr2=0x7f86040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8604065430 0x7f86040678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8608032430 tx=0x7f8608009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8618101b70 msgr2=0x7f861810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8618101b70 0x7f861810e4e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f860000c830 tx=0x7f8600004430 comp rx=0 tx=0).stop 2024-06-06T08:27:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 shutdown_connections 2024-06-06T08:27:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8604065430 0x7f86040678e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8618101b70 0x7f861810e4e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 --2- 172.21.15.86:0/4067161191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86181011d0 0x7f861810dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 >> 172.21.15.86:0/4067161191 conn(0x7f86180f6dc0 msgr2=0x7f86180f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 shutdown_connections 2024-06-06T08:27:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:39.876+0000 7f861cb78640 1 -- 172.21.15.86:0/4067161191 wait complete. 2024-06-06T08:27:40.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:40 smithi086 ceph-mon[29114]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:40.151 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:27:40.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:40 smithi154 ceph-mon[35428]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:40.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:40 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:27:40.484 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-06-06T08:27:38.691741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788036,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682876,"statfs":{"total":575970213888,"available":574139265024,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002242"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-06T08:26:45.458910+0000","last_change":"2024-06-06T08:26:20.024006+0000","last_active":"2024-06-06T08:26:45.458910+0000","last_peered":"2024-06-06T08:26:45.458910+0000","last_clean":"2024-06-06T08:26:45.458910+0000","last_became_active":"2024-06-06T08:26:20.023753+0000","last_became_peered":"2024-06-06T08:26:20.023753+0000","last_unstale":"2024-06-06T08:26:45.458910+0000","last_undegraded":"2024-06-06T08:26:45.458910+0000","last_fullsized":"2024-06-06T08:26:45.458910+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-06T08:25:32.961935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-06T08:25:32.961935+0000","last_clean_scrub_stamp":"2024-06-06T08:25:32.961935+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-07T13:13:12.879348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 08:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jun 6 08:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Jun 6 08:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.378}]},{"osd":3,"last update":"Thu Jun 6 08:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.25800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 08:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jun 6 08:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.37}]},{"osd":2,"last update":"Thu Jun 6 08:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.441},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.432}]},{"osd":4,"last update":"Thu Jun 6 08:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.315}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.33200000000000002,"15min":0.33700000000000002},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.316,"5min":0.33700000000000002,"15min":0.34100000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.32600000000000001}]},{"osd":1,"last update":"Thu Jun 6 08:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.33900000000000002,"15min":0.34399999999999997},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.44800000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.246},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.35799999999999998,"15min":0.36099999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.309}]},{"osd":3,"last update":"Thu Jun 6 08:26:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jun 6 08:27:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.504},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 08:27:08 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.33000000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.33900000000000002,"15min":0.33600000000000002},"min":{"1min":0.26600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.495,"5min":0.499,"15min":0.499},"last":0.251}]},{"osd":2,"last update":"Thu Jun 6 08:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.46100000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.379,"15min":0.38700000000000001},"min":{"1min":0.255,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.47199999999999998,"5min":0.495,"15min":0.495},"last":0.27500000000000002}]},{"osd":3,"last update":"Thu Jun 6 08:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Thu Jun 6 08:27:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 6 08:27:09 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.313,"15min":0.311},"min":{"1min":0.22900000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.45700000000000002,"5min":0.496,"15min":0.496},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.317},"min":{"1min":0.24199999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.375}]},{"osd":2,"last update":"Thu Jun 6 08:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.34100000000000003}]},{"osd":3,"last update":"Thu Jun 6 08:26:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.754},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jun 6 08:27:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.73899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-06T08:27:40.485 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-06T08:27:40.485 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-06T08:27:40.485 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-06T08:27:40.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:27:40.736 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:41.122 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:41 smithi086 ceph-mon[29114]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T08:27:41.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:41 smithi154 ceph-mon[35428]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T08:27:42.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.239+0000 7f05e8831640 1 -- 172.21.15.86:0/1946717226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e41022c0 msgr2=0x7f05e4102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:42.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.239+0000 7f05e8831640 1 --2- 172.21.15.86:0/1946717226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e41022c0 0x7f05e4102690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f05d40048d0 tx=0x7f05d4009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.239+0000 7f05e8831640 1 -- 172.21.15.86:0/1946717226 shutdown_connections 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.239+0000 7f05e8831640 1 --2- 172.21.15.86:0/1946717226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05e4102c60 0x7f05e40fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.239+0000 7f05e8831640 1 --2- 172.21.15.86:0/1946717226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e41022c0 0x7f05e4102690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.239+0000 7f05e8831640 1 -- 172.21.15.86:0/1946717226 >> 172.21.15.86:0/1946717226 conn(0x7f05e40f7e50 msgr2=0x7f05e40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.239+0000 7f05e8831640 1 -- 172.21.15.86:0/1946717226 shutdown_connections 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.239+0000 7f05e8831640 1 -- 172.21.15.86:0/1946717226 wait complete. 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.239+0000 7f05e8831640 1 Processor -- start 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05e8831640 1 -- start start 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05e8831640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05e41022c0 0x7f05e4113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05e8831640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e4102c60 0x7f05e4113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05e8831640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e4113e70 con 0x7f05e4102c60 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05e8831640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e410d2b0 con 0x7f05e41022c0 2024-06-06T08:27:42.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05dbfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e4102c60 0x7f05e4113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05dbfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e4102c60 0x7f05e4113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51844/0 (socket says 172.21.15.86:51844) 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05dbfff640 1 -- 172.21.15.86:0/516940950 learned_addr learned my addr 172.21.15.86:0/516940950 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05e259c640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05e41022c0 0x7f05e4113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05dbfff640 1 -- 172.21.15.86:0/516940950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05e41022c0 msgr2=0x7f05e4113330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05dbfff640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05e41022c0 0x7f05e4113330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05dbfff640 1 -- 172.21.15.86:0/516940950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d4005040 con 0x7f05e4102c60 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05dbfff640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e4102c60 0x7f05e4113870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f05cc002a50 tx=0x7f05cc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05daffd640 1 -- 172.21.15.86:0/516940950 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05cc007c30 con 0x7f05e4102c60 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05daffd640 1 -- 172.21.15.86:0/516940950 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05cc00f040 con 0x7f05e4102c60 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e410d590 con 0x7f05e4102c60 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.240+0000 7f05daffd640 1 -- 172.21.15.86:0/516940950 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05cc00dad0 con 0x7f05e4102c60 2024-06-06T08:27:42.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.241+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e410db30 con 0x7f05e4102c60 2024-06-06T08:27:42.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.241+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e40fda30 con 0x7f05e4102c60 2024-06-06T08:27:42.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.242+0000 7f05daffd640 1 -- 172.21.15.86:0/516940950 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f05cc00dc70 con 0x7f05e4102c60 2024-06-06T08:27:42.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.242+0000 7f05daffd640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f05d0065430 0x7f05d00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:42.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.242+0000 7f05daffd640 1 -- 172.21.15.86:0/516940950 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f05cc081bb0 con 0x7f05e4102c60 2024-06-06T08:27:42.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.242+0000 7f05e259c640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f05d0065430 0x7f05d00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:42.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.243+0000 7f05e259c640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f05d0065430 0x7f05d00678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f05d4009f80 tx=0x7f05d4035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:42.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.246+0000 7f05daffd640 1 -- 172.21.15.86:0/516940950 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f05cc04cdf0 con 0x7f05e4102c60 2024-06-06T08:27:42.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:42 smithi086 ceph-mon[29114]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:42.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:42 smithi154 ceph-mon[35428]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:42.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.445+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f05e4109420 con 0x7f05e4102c60 2024-06-06T08:27:42.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.446+0000 7f05daffd640 1 -- 172.21.15.86:0/516940950 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f05cc088080 con 0x7f05e4102c60 2024-06-06T08:27:42.445 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:27:42.445 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f05d0065430 msgr2=0x7f05d00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f05d0065430 0x7f05d00678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f05d4009f80 tx=0x7f05d4035040 comp rx=0 tx=0).stop 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e4102c60 msgr2=0x7f05e4113870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e4102c60 0x7f05e4113870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f05cc002a50 tx=0x7f05cc004600 comp rx=0 tx=0).stop 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 shutdown_connections 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f05d0065430 0x7f05d00678e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05e4102c60 0x7f05e4113870 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 --2- 172.21.15.86:0/516940950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05e41022c0 0x7f05e4113330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 >> 172.21.15.86:0/516940950 conn(0x7f05e40f7e50 msgr2=0x7f05e40f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 shutdown_connections 2024-06-06T08:27:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:42.448+0000 7f05e8831640 1 -- 172.21.15.86:0/516940950 wait complete. 2024-06-06T08:27:43.304 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:43 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/516940950' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:27:43.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:43 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/516940950' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:27:44.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:44 smithi086 ceph-mon[29114]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:44.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:44 smithi154 ceph-mon[35428]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:46.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:27:46.315 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:46.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:46 smithi086 ceph-mon[29114]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:46.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:46 smithi154 ceph-mon[35428]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:47.324 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:47 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:27:47.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:47 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:27:47.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 -- 172.21.15.86:0/4176379152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 msgr2=0x7f382c1026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 --2- 172.21.15.86:0/4176379152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 0x7f382c1026a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f381c0048d0 tx=0x7f381c009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 -- 172.21.15.86:0/4176379152 shutdown_connections 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 --2- 172.21.15.86:0/4176379152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382c102c70 0x7f382c0fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 --2- 172.21.15.86:0/4176379152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 0x7f382c1026a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 -- 172.21.15.86:0/4176379152 >> 172.21.15.86:0/4176379152 conn(0x7f382c0f7e60 msgr2=0x7f382c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 -- 172.21.15.86:0/4176379152 shutdown_connections 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 -- 172.21.15.86:0/4176379152 wait complete. 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 Processor -- start 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.809+0000 7f38328ba640 1 -- start start 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f38328ba640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 0x7f382c112fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f38328ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382c102c70 0x7f382c113510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:47.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f38328ba640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c113b10 con 0x7f382c1022d0 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f38328ba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c10cf50 con 0x7f382c102c70 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f382bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 0x7f382c112fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f382bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 0x7f382c112fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54108/0 (socket says 172.21.15.86:54108) 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f382bfff640 1 -- 172.21.15.86:0/953313324 learned_addr learned my addr 172.21.15.86:0/953313324 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f382b7fe640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382c102c70 0x7f382c113510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f382bfff640 1 -- 172.21.15.86:0/953313324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382c102c70 msgr2=0x7f382c113510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f382bfff640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382c102c70 0x7f382c113510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f382bfff640 1 -- 172.21.15.86:0/953313324 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c005040 con 0x7f382c1022d0 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f382bfff640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 0x7f382c112fd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f381c003080 tx=0x7f381c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f3828ff9640 1 -- 172.21.15.86:0/953313324 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f381c038070 con 0x7f382c1022d0 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382c10d1d0 con 0x7f382c1022d0 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f3828ff9640 1 -- 172.21.15.86:0/953313324 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f381c008ee0 con 0x7f382c1022d0 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.810+0000 7f3828ff9640 1 -- 172.21.15.86:0/953313324 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f381c03d870 con 0x7f382c1022d0 2024-06-06T08:27:47.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.811+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382c10d710 con 0x7f382c1022d0 2024-06-06T08:27:47.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.812+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382c0fda40 con 0x7f382c1022d0 2024-06-06T08:27:47.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.812+0000 7f3828ff9640 1 -- 172.21.15.86:0/953313324 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f381c030020 con 0x7f382c1022d0 2024-06-06T08:27:47.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.812+0000 7f3828ff9640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3814065430 0x7f38140678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:47.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.812+0000 7f3828ff9640 1 -- 172.21.15.86:0/953313324 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f381c0a21b0 con 0x7f382c1022d0 2024-06-06T08:27:47.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.812+0000 7f382b7fe640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3814065430 0x7f38140678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:47.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.813+0000 7f382b7fe640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3814065430 0x7f38140678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f382c10e890 tx=0x7f3820004330 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:47.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:47.816+0000 7f3828ff9640 1 -- 172.21.15.86:0/953313324 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f381c0706e0 con 0x7f382c1022d0 2024-06-06T08:27:48.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.012+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f382c109410 con 0x7f382c1022d0 2024-06-06T08:27:48.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.013+0000 7f3828ff9640 1 -- 172.21.15.86:0/953313324 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f381c073d00 con 0x7f382c1022d0 2024-06-06T08:27:48.013 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:27:48.013 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:27:48.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.015+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3814065430 msgr2=0x7f38140678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.015+0000 7f38328ba640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3814065430 0x7f38140678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f382c10e890 tx=0x7f3820004330 comp rx=0 tx=0).stop 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.015+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 msgr2=0x7f382c112fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.015+0000 7f38328ba640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 0x7f382c112fd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f381c003080 tx=0x7f381c009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.016+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 shutdown_connections 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.016+0000 7f38328ba640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382c102c70 0x7f382c113510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.016+0000 7f38328ba640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3814065430 0x7f38140678e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.016+0000 7f38328ba640 1 --2- 172.21.15.86:0/953313324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f382c1022d0 0x7f382c112fd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.016+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 >> 172.21.15.86:0/953313324 conn(0x7f382c0f7e60 msgr2=0x7f382c0f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.016+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 shutdown_connections 2024-06-06T08:27:48.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:48.016+0000 7f38328ba640 1 -- 172.21.15.86:0/953313324 wait complete. 2024-06-06T08:27:48.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:48 smithi086 ceph-mon[29114]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:48.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:27:48.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:27:48.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:27:48.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:48 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/953313324' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:27:48.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:48 smithi154 ceph-mon[35428]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:48.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:27:48.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:27:48.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:27:48.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:48 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/953313324' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:27:50.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:50 smithi086 ceph-mon[29114]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:50 smithi154 ceph-mon[35428]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:51.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:27:51.833 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:52.256 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:52 smithi086 ceph-mon[29114]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:52.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:52 smithi154 ceph-mon[35428]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.363+0000 7f95dc861640 1 -- 172.21.15.86:0/742245336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 msgr2=0x7f95d80fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.363+0000 7f95dc861640 1 --2- 172.21.15.86:0/742245336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 0x7f95d80fc2f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f95c00048d0 tx=0x7f95c0009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.363+0000 7f95dc861640 1 -- 172.21.15.86:0/742245336 shutdown_connections 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.363+0000 7f95dc861640 1 --2- 172.21.15.86:0/742245336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 0x7f95d80fc2f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.363+0000 7f95dc861640 1 --2- 172.21.15.86:0/742245336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d81022c0 0x7f95d8102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.363+0000 7f95dc861640 1 -- 172.21.15.86:0/742245336 >> 172.21.15.86:0/742245336 conn(0x7f95d80f7e50 msgr2=0x7f95d80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.363+0000 7f95dc861640 1 -- 172.21.15.86:0/742245336 shutdown_connections 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.363+0000 7f95dc861640 1 -- 172.21.15.86:0/742245336 wait complete. 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95dc861640 1 Processor -- start 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95dc861640 1 -- start start 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95dc861640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d81022c0 0x7f95d8113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:53.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95dc861640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 0x7f95d81138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95dc861640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d8113eb0 con 0x7f95d8102c60 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95dc861640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d810d2f0 con 0x7f95d81022c0 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95d5d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 0x7f95d81138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95d5d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 0x7f95d81138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54138/0 (socket says 172.21.15.86:54138) 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95d5d9b640 1 -- 172.21.15.86:0/1074676795 learned_addr learned my addr 172.21.15.86:0/1074676795 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95d659c640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d81022c0 0x7f95d8113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95d5d9b640 1 -- 172.21.15.86:0/1074676795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d81022c0 msgr2=0x7f95d8113370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95d5d9b640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d81022c0 0x7f95d8113370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95d5d9b640 1 -- 172.21.15.86:0/1074676795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c0005040 con 0x7f95d8102c60 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95d5d9b640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 0x7f95d81138b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f95c00043d0 tx=0x7f95c0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.364+0000 7f95ceffd640 1 -- 172.21.15.86:0/1074676795 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95c0038070 con 0x7f95d8102c60 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.365+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d810d570 con 0x7f95d8102c60 2024-06-06T08:27:53.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.365+0000 7f95ceffd640 1 -- 172.21.15.86:0/1074676795 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95c0032970 con 0x7f95d8102c60 2024-06-06T08:27:53.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.365+0000 7f95ceffd640 1 -- 172.21.15.86:0/1074676795 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95c003da60 con 0x7f95d8102c60 2024-06-06T08:27:53.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.365+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d810dab0 con 0x7f95d8102c60 2024-06-06T08:27:53.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.366+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d80fda30 con 0x7f95d8102c60 2024-06-06T08:27:53.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.366+0000 7f95ceffd640 1 -- 172.21.15.86:0/1074676795 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f95c0030020 con 0x7f95d8102c60 2024-06-06T08:27:53.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.366+0000 7f95ceffd640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f95c4065430 0x7f95c40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:53.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.366+0000 7f95ceffd640 1 -- 172.21.15.86:0/1074676795 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f95c00a25c0 con 0x7f95d8102c60 2024-06-06T08:27:53.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.366+0000 7f95d659c640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f95c4065430 0x7f95c40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:53.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.367+0000 7f95d659c640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f95c4065430 0x7f95c40678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f95d0004ed0 tx=0x7f95d000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:53.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.371+0000 7f95ceffd640 1 -- 172.21.15.86:0/1074676795 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f95c0070a70 con 0x7f95d8102c60 2024-06-06T08:27:53.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.568+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f95d8109420 con 0x7f95d8102c60 2024-06-06T08:27:53.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.569+0000 7f95ceffd640 1 -- 172.21.15.86:0/1074676795 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f95c0074090 con 0x7f95d8102c60 2024-06-06T08:27:53.568 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:27:53.568 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:27:53.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f95c4065430 msgr2=0x7f95c40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:53.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f95c4065430 0x7f95c40678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f95d0004ed0 tx=0x7f95d000b040 comp rx=0 tx=0).stop 2024-06-06T08:27:53.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 msgr2=0x7f95d81138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:53.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 0x7f95d81138b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f95c00043d0 tx=0x7f95c0009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:53.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 shutdown_connections 2024-06-06T08:27:53.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f95c4065430 0x7f95c40678e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:53.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d8102c60 0x7f95d81138b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 --2- 172.21.15.86:0/1074676795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d81022c0 0x7f95d8113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 >> 172.21.15.86:0/1074676795 conn(0x7f95d80f7e50 msgr2=0x7f95d80f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 shutdown_connections 2024-06-06T08:27:53.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:53.571+0000 7f95dc861640 1 -- 172.21.15.86:0/1074676795 wait complete. 2024-06-06T08:27:54.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:54 smithi086 ceph-mon[29114]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:54.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:54 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1074676795' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:27:54.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:54 smithi154 ceph-mon[35428]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:54.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:54 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1074676795' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:27:55.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:55 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:27:55.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:55 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:27:56.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:56 smithi154 ceph-mon[35428]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:56.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:56 smithi086 ceph-mon[29114]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:57.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:27:57.447 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:27:58.164 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:27:58 smithi086 ceph-mon[29114]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:58.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:27:58 smithi154 ceph-mon[35428]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:27:59.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.978+0000 7f311d021640 1 -- 172.21.15.86:0/3305356335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31181022a0 msgr2=0x7f3118102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.978+0000 7f311d021640 1 --2- 172.21.15.86:0/3305356335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31181022a0 0x7f3118102670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f31080048d0 tx=0x7f3108009da0 comp rx=0 tx=0).stop 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.978+0000 7f311d021640 1 -- 172.21.15.86:0/3305356335 shutdown_connections 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.978+0000 7f311d021640 1 --2- 172.21.15.86:0/3305356335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3118102c40 0x7f31180fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.978+0000 7f311d021640 1 --2- 172.21.15.86:0/3305356335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31181022a0 0x7f3118102670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.978+0000 7f311d021640 1 -- 172.21.15.86:0/3305356335 >> 172.21.15.86:0/3305356335 conn(0x7f31180f7e50 msgr2=0x7f31180fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.978+0000 7f311d021640 1 -- 172.21.15.86:0/3305356335 shutdown_connections 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.978+0000 7f311d021640 1 -- 172.21.15.86:0/3305356335 wait complete. 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311d021640 1 Processor -- start 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311d021640 1 -- start start 2024-06-06T08:27:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311d021640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31181022a0 0x7f311806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311d021640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3118102c40 0x7f311806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311d021640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31180701f0 con 0x7f3118102c40 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311d021640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3118070360 con 0x7f31181022a0 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3118102c40 0x7f311806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3118102c40 0x7f311806fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:42736/0 (socket says 172.21.15.86:42736) 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311659c640 1 -- 172.21.15.86:0/403365995 learned_addr learned my addr 172.21.15.86:0/403365995 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f3116d9d640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31181022a0 0x7f311806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311659c640 1 -- 172.21.15.86:0/403365995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31181022a0 msgr2=0x7f311806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311659c640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31181022a0 0x7f311806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311659c640 1 -- 172.21.15.86:0/403365995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3108005040 con 0x7f3118102c40 2024-06-06T08:27:59.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.979+0000 7f311659c640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3118102c40 0x7f311806fcb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f310c00c810 tx=0x7f310c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.980+0000 7f30ff7fe640 1 -- 172.21.15.86:0/403365995 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f310c00a7a0 con 0x7f3118102c40 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.980+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31180705f0 con 0x7f3118102c40 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.980+0000 7f30ff7fe640 1 -- 172.21.15.86:0/403365995 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f310c00add0 con 0x7f3118102c40 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.980+0000 7f30ff7fe640 1 -- 172.21.15.86:0/403365995 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f310c0115c0 con 0x7f3118102c40 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.980+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31181196e0 con 0x7f3118102c40 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.981+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31180fda10 con 0x7f3118102c40 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.981+0000 7f30ff7fe640 1 -- 172.21.15.86:0/403365995 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f310c012420 con 0x7f3118102c40 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.981+0000 7f30ff7fe640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3100065430 0x7f31000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.982+0000 7f30ff7fe640 1 -- 172.21.15.86:0/403365995 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f310c082970 con 0x7f3118102c40 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.982+0000 7f3116d9d640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3100065430 0x7f31000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.982+0000 7f3116d9d640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3100065430 0x7f31000678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3108009f80 tx=0x7f3108035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:27:59.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:58.986+0000 7f30ff7fe640 1 -- 172.21.15.86:0/403365995 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f310c050e20 con 0x7f3118102c40 2024-06-06T08:27:59.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.185+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f31180fba90 con 0x7f3118102c40 2024-06-06T08:27:59.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.185+0000 7f30ff7fe640 1 -- 172.21.15.86:0/403365995 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f310c054440 con 0x7f3118102c40 2024-06-06T08:27:59.185 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:27:59.185 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:27:59.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3100065430 msgr2=0x7f31000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:59.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3100065430 0x7f31000678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3108009f80 tx=0x7f3108035040 comp rx=0 tx=0).stop 2024-06-06T08:27:59.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3118102c40 msgr2=0x7f311806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:27:59.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3118102c40 0x7f311806fcb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f310c00c810 tx=0x7f310c0043c0 comp rx=0 tx=0).stop 2024-06-06T08:27:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 shutdown_connections 2024-06-06T08:27:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3100065430 0x7f31000678e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3118102c40 0x7f311806fcb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 --2- 172.21.15.86:0/403365995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31181022a0 0x7f311806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:27:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 >> 172.21.15.86:0/403365995 conn(0x7f31180f7e50 msgr2=0x7f31180f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:27:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 shutdown_connections 2024-06-06T08:27:59.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:27:59.187+0000 7f311d021640 1 -- 172.21.15.86:0/403365995 wait complete. 2024-06-06T08:28:00.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:00 smithi154 ceph-mon[35428]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:00.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:00 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/403365995' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:00.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:00 smithi086 ceph-mon[29114]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:00.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:00 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/403365995' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:02.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:02 smithi154 ceph-mon[35428]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:02.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:02 smithi086 ceph-mon[29114]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:02.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:04.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:04 smithi086 ceph-mon[29114]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:04 smithi154 ceph-mon[35428]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.541+0000 7ff2a27b8640 1 -- 172.21.15.86:0/70596253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c101b50 msgr2=0x7ff29c0fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.541+0000 7ff2a27b8640 1 --2- 172.21.15.86:0/70596253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c101b50 0x7ff29c0fb1e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff2900048d0 tx=0x7ff290009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.541+0000 7ff2a27b8640 1 -- 172.21.15.86:0/70596253 shutdown_connections 2024-06-06T08:28:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.541+0000 7ff2a27b8640 1 --2- 172.21.15.86:0/70596253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c101b50 0x7ff29c0fb1e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.541+0000 7ff2a27b8640 1 --2- 172.21.15.86:0/70596253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29c1011b0 0x7ff29c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.541+0000 7ff2a27b8640 1 -- 172.21.15.86:0/70596253 >> 172.21.15.86:0/70596253 conn(0x7ff29c0f6d60 msgr2=0x7ff29c0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.541+0000 7ff2a27b8640 1 -- 172.21.15.86:0/70596253 shutdown_connections 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.541+0000 7ff2a27b8640 1 -- 172.21.15.86:0/70596253 wait complete. 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.541+0000 7ff2a27b8640 1 Processor -- start 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff2a27b8640 1 -- start start 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff2a27b8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c1011b0 0x7ff29c10e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff2a27b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29c101b50 0x7ff29c10e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff2a27b8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c10ec40 con 0x7ff29c1011b0 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff2a27b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c10edb0 con 0x7ff29c101b50 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff29bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c1011b0 0x7ff29c10e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff29bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c1011b0 0x7ff29c10e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:42762/0 (socket says 172.21.15.86:42762) 2024-06-06T08:28:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff29bfff640 1 -- 172.21.15.86:0/3897177084 learned_addr learned my addr 172.21.15.86:0/3897177084 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff29b7fe640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29c101b50 0x7ff29c10e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff29bfff640 1 -- 172.21.15.86:0/3897177084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29c101b50 msgr2=0x7ff29c10e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff29bfff640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29c101b50 0x7ff29c10e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff29bfff640 1 -- 172.21.15.86:0/3897177084 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff290005040 con 0x7ff29c1011b0 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.542+0000 7ff29b7fe640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29c101b50 0x7ff29c10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.543+0000 7ff29bfff640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c1011b0 0x7ff29c10e1c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00b790 tx=0x7ff28c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.543+0000 7ff298ff9640 1 -- 172.21.15.86:0/3897177084 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28c00c700 con 0x7ff29c1011b0 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.543+0000 7ff298ff9640 1 -- 172.21.15.86:0/3897177084 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff28c00cd30 con 0x7ff29c1011b0 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.543+0000 7ff298ff9640 1 -- 172.21.15.86:0/3897177084 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28c013370 con 0x7ff29c1011b0 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.543+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29c1170f0 con 0x7ff29c1011b0 2024-06-06T08:28:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.543+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29c117610 con 0x7ff29c1011b0 2024-06-06T08:28:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.544+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29c0fc920 con 0x7ff29c1011b0 2024-06-06T08:28:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.544+0000 7ff298ff9640 1 -- 172.21.15.86:0/3897177084 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff28c012070 con 0x7ff29c1011b0 2024-06-06T08:28:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.545+0000 7ff298ff9640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff284065430 0x7ff2840678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.545+0000 7ff298ff9640 1 -- 172.21.15.86:0/3897177084 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff28c0820a0 con 0x7ff29c1011b0 2024-06-06T08:28:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.545+0000 7ff29b7fe640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff284065430 0x7ff2840678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.545+0000 7ff29b7fe640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff284065430 0x7ff2840678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff290009f80 tx=0x7ff290009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:04.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.549+0000 7ff298ff9640 1 -- 172.21.15.86:0/3897177084 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff28c054060 con 0x7ff29c1011b0 2024-06-06T08:28:04.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.748+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff29c0fa9a0 con 0x7ff29c1011b0 2024-06-06T08:28:04.747 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:04.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.748+0000 7ff298ff9640 1 -- 172.21.15.86:0/3897177084 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff28c053bf0 con 0x7ff29c1011b0 2024-06-06T08:28:04.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff284065430 msgr2=0x7ff2840678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:04.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff284065430 0x7ff2840678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff290009f80 tx=0x7ff290009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:04.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c1011b0 msgr2=0x7ff29c10e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:04.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c1011b0 0x7ff29c10e1c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00b790 tx=0x7ff28c0091d0 comp rx=0 tx=0).stop 2024-06-06T08:28:04.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 shutdown_connections 2024-06-06T08:28:04.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29c101b50 0x7ff29c10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:04.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff284065430 0x7ff2840678e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:04.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 --2- 172.21.15.86:0/3897177084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c1011b0 0x7ff29c10e1c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:04.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 >> 172.21.15.86:0/3897177084 conn(0x7ff29c0f6d60 msgr2=0x7ff29c0f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:04.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 shutdown_connections 2024-06-06T08:28:04.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:04.750+0000 7ff2a27b8640 1 -- 172.21.15.86:0/3897177084 wait complete. 2024-06-06T08:28:05.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:05 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3897177084' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:05.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:05 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3897177084' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:06.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:06 smithi154 ceph-mon[35428]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:06.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:06 smithi086 ceph-mon[29114]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:08.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:08.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:08 smithi154 ceph-mon[35428]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:08.553 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:08.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:08 smithi086 ceph-mon[29114]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 -- 172.21.15.86:0/4047251989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 msgr2=0x7f2794102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 --2- 172.21.15.86:0/4047251989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 0x7f2794102740 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f278c0048d0 tx=0x7f278c009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 -- 172.21.15.86:0/4047251989 shutdown_connections 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 --2- 172.21.15.86:0/4047251989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27940fa9c0 0x7f27940fae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 --2- 172.21.15.86:0/4047251989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 0x7f2794102740 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 -- 172.21.15.86:0/4047251989 >> 172.21.15.86:0/4047251989 conn(0x7f27940709e0 msgr2=0x7f2794070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 -- 172.21.15.86:0/4047251989 shutdown_connections 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 -- 172.21.15.86:0/4047251989 wait complete. 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 Processor -- start 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.319+0000 7f2798bd9640 1 -- start start 2024-06-06T08:28:10.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2798bd9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27940fa9c0 0x7f2794113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2798bd9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 0x7f2794113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2798bd9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2794113b40 con 0x7f2794102370 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2798bd9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279410cf80 con 0x7f27940fa9c0 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2791d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 0x7f2794113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2791d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 0x7f2794113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52688/0 (socket says 172.21.15.86:52688) 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2791d9b640 1 -- 172.21.15.86:0/1758261346 learned_addr learned my addr 172.21.15.86:0/1758261346 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f279259c640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27940fa9c0 0x7f2794113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2791d9b640 1 -- 172.21.15.86:0/1758261346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27940fa9c0 msgr2=0x7f2794113000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2791d9b640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27940fa9c0 0x7f2794113000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2791d9b640 1 -- 172.21.15.86:0/1758261346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c005040 con 0x7f2794102370 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f279259c640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27940fa9c0 0x7f2794113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2791d9b640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 0x7f2794113540 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f278800cae0 tx=0x7f2788007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2782ffd640 1 -- 172.21.15.86:0/1758261346 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f278800ab00 con 0x7f2794102370 2024-06-06T08:28:10.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2782ffd640 1 -- 172.21.15.86:0/1758261346 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f278800ac60 con 0x7f2794102370 2024-06-06T08:28:10.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279410d260 con 0x7f2794102370 2024-06-06T08:28:10.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.320+0000 7f2782ffd640 1 -- 172.21.15.86:0/1758261346 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27880116f0 con 0x7f2794102370 2024-06-06T08:28:10.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.321+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279410d800 con 0x7f2794102370 2024-06-06T08:28:10.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.321+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2794065570 con 0x7f2794102370 2024-06-06T08:28:10.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.322+0000 7f2782ffd640 1 -- 172.21.15.86:0/1758261346 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f2788012440 con 0x7f2794102370 2024-06-06T08:28:10.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.322+0000 7f2782ffd640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f277c065430 0x7f277c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:10.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.323+0000 7f279259c640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f277c065430 0x7f277c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:10.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.323+0000 7f2782ffd640 1 -- 172.21.15.86:0/1758261346 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2788082dc0 con 0x7f2794102370 2024-06-06T08:28:10.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.323+0000 7f279259c640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f277c065430 0x7f277c0678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f278c009f80 tx=0x7f278c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:10.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.327+0000 7f2782ffd640 1 -- 172.21.15.86:0/1758261346 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2788051270 con 0x7f2794102370 2024-06-06T08:28:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:10 smithi154 ceph-mon[35428]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:10 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:28:10.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.522+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f279410e6a0 con 0x7f2794102370 2024-06-06T08:28:10.522 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:10 smithi086 ceph-mon[29114]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:10.522 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:10 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:28:10.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.523+0000 7f2782ffd640 1 -- 172.21.15.86:0/1758261346 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2788054890 con 0x7f2794102370 2024-06-06T08:28:10.523 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:10.523 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:10.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f277c065430 msgr2=0x7f277c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:10.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f277c065430 0x7f277c0678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f278c009f80 tx=0x7f278c035040 comp rx=0 tx=0).stop 2024-06-06T08:28:10.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 msgr2=0x7f2794113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:10.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 0x7f2794113540 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f278800cae0 tx=0x7f2788007e50 comp rx=0 tx=0).stop 2024-06-06T08:28:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 shutdown_connections 2024-06-06T08:28:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f277c065430 0x7f277c0678e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2794102370 0x7f2794113540 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 --2- 172.21.15.86:0/1758261346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27940fa9c0 0x7f2794113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 >> 172.21.15.86:0/1758261346 conn(0x7f27940709e0 msgr2=0x7f27940f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 shutdown_connections 2024-06-06T08:28:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:10.525+0000 7f2798bd9640 1 -- 172.21.15.86:0/1758261346 wait complete. 2024-06-06T08:28:11.154 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:11 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1758261346' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:11.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:11 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1758261346' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:12.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:12 smithi154 ceph-mon[35428]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:12.426 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:12 smithi086 ceph-mon[29114]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:14.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:14.410 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:14.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:14 smithi154 ceph-mon[35428]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:14.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:14 smithi086 ceph-mon[29114]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:15.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.839+0000 7f7f4142a640 1 -- 172.21.15.86:0/1126642504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c1011a0 msgr2=0x7f7f3c101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.839+0000 7f7f4142a640 1 --2- 172.21.15.86:0/1126642504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c1011a0 0x7f7f3c101570 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0048d0 tx=0x7f7f2c009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.840+0000 7f7f4142a640 1 -- 172.21.15.86:0/1126642504 shutdown_connections 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.840+0000 7f7f4142a640 1 --2- 172.21.15.86:0/1126642504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c101b40 0x7f7f3c0fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.840+0000 7f7f4142a640 1 --2- 172.21.15.86:0/1126642504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c1011a0 0x7f7f3c101570 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.840+0000 7f7f4142a640 1 -- 172.21.15.86:0/1126642504 >> 172.21.15.86:0/1126642504 conn(0x7f7f3c0f6d90 msgr2=0x7f7f3c0f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.840+0000 7f7f4142a640 1 -- 172.21.15.86:0/1126642504 shutdown_connections 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.840+0000 7f7f4142a640 1 -- 172.21.15.86:0/1126642504 wait complete. 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.840+0000 7f7f4142a640 1 Processor -- start 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.840+0000 7f7f4142a640 1 -- start start 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f4142a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c1011a0 0x7f7f3c110e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f4142a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c101b40 0x7f7f3c111380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f4142a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c111a10 con 0x7f7f3c101b40 2024-06-06T08:28:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f4142a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c10adc0 con 0x7f7f3c1011a0 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f327fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c101b40 0x7f7f3c111380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f327fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c101b40 0x7f7f3c111380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52708/0 (socket says 172.21.15.86:52708) 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f327fc640 1 -- 172.21.15.86:0/3077539738 learned_addr learned my addr 172.21.15.86:0/3077539738 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f3affd640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c1011a0 0x7f7f3c110e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f327fc640 1 -- 172.21.15.86:0/3077539738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c1011a0 msgr2=0x7f7f3c110e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f327fc640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c1011a0 0x7f7f3c110e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f327fc640 1 -- 172.21.15.86:0/3077539738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c005040 con 0x7f7f3c101b40 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f327fc640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c101b40 0x7f7f3c111380 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400b790 tx=0x7f7f240091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f33fff640 1 -- 172.21.15.86:0/3077539738 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2400c700 con 0x7f7f3c101b40 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f33fff640 1 -- 172.21.15.86:0/3077539738 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f2400cd30 con 0x7f7f3c101b40 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c10b0a0 con 0x7f7f3c101b40 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f33fff640 1 -- 172.21.15.86:0/3077539738 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f24013370 con 0x7f7f3c101b40 2024-06-06T08:28:15.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.841+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c10b5c0 con 0x7f7f3c101b40 2024-06-06T08:28:15.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.842+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3c0fc910 con 0x7f7f3c101b40 2024-06-06T08:28:15.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.843+0000 7f7f33fff640 1 -- 172.21.15.86:0/3077539738 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f7f24012070 con 0x7f7f3c101b40 2024-06-06T08:28:15.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.843+0000 7f7f33fff640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7f28067640 0x7f7f28069af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:15.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.843+0000 7f7f33fff640 1 -- 172.21.15.86:0/3077539738 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7f24081dd0 con 0x7f7f3c101b40 2024-06-06T08:28:15.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.843+0000 7f7f3affd640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7f28067640 0x7f7f28069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:15.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.844+0000 7f7f3affd640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7f28067640 0x7f7f28069af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c032430 tx=0x7f7f2c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:15.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:15.848+0000 7f7f33fff640 1 -- 172.21.15.86:0/3077539738 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7f24050280 con 0x7f7f3c101b40 2024-06-06T08:28:16.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.043+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7f3c107280 con 0x7f7f3c101b40 2024-06-06T08:28:16.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.043+0000 7f7f33fff640 1 -- 172.21.15.86:0/3077539738 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7f240538a0 con 0x7f7f3c101b40 2024-06-06T08:28:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:16.043 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:16.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7f28067640 msgr2=0x7f7f28069af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:16.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7f28067640 0x7f7f28069af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c032430 tx=0x7f7f2c035040 comp rx=0 tx=0).stop 2024-06-06T08:28:16.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c101b40 msgr2=0x7f7f3c111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:16.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c101b40 0x7f7f3c111380 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400b790 tx=0x7f7f240091d0 comp rx=0 tx=0).stop 2024-06-06T08:28:16.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 shutdown_connections 2024-06-06T08:28:16.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7f28067640 0x7f7f28069af0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:16.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f3c101b40 0x7f7f3c111380 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:16.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 --2- 172.21.15.86:0/3077539738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f3c1011a0 0x7f7f3c110e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:16.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 >> 172.21.15.86:0/3077539738 conn(0x7f7f3c0f6d90 msgr2=0x7f7f3c0f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:16.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 shutdown_connections 2024-06-06T08:28:16.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:16.045+0000 7f7f4142a640 1 -- 172.21.15.86:0/3077539738 wait complete. 2024-06-06T08:28:16.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:16 smithi086 ceph-mon[29114]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:16.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:16 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3077539738' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:16.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:16 smithi154 ceph-mon[35428]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:16.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:16 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3077539738' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:18.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:18 smithi154 ceph-mon[35428]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:18.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:18 smithi086 ceph-mon[29114]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:19.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:19.978 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:20.369 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:20 smithi086 ceph-mon[29114]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:20.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:20 smithi154 ceph-mon[35428]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:21.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.528+0000 7fa7a73d2640 1 -- 172.21.15.86:0/3704253798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 msgr2=0x7fa7a0101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:21.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.528+0000 7fa7a73d2640 1 --2- 172.21.15.86:0/3704253798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 0x7fa7a0101580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa7900048d0 tx=0x7fa790009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:21.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 -- 172.21.15.86:0/3704253798 shutdown_connections 2024-06-06T08:28:21.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 --2- 172.21.15.86:0/3704253798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7a0101b50 0x7fa7a00fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:21.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 --2- 172.21.15.86:0/3704253798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 0x7fa7a0101580 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:21.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 -- 172.21.15.86:0/3704253798 >> 172.21.15.86:0/3704253798 conn(0x7fa7a00f6d60 msgr2=0x7fa7a00f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 -- 172.21.15.86:0/3704253798 shutdown_connections 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 -- 172.21.15.86:0/3704253798 wait complete. 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 Processor -- start 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 -- start start 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 0x7fa7a010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7a0101b50 0x7fa7a010e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a010eca0 con 0x7fa7a01011b0 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.529+0000 7fa7a73d2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a010ee10 con 0x7fa7a0101b50 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a516e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 0x7fa7a010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a516e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 0x7fa7a010e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43478/0 (socket says 172.21.15.86:43478) 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a516e640 1 -- 172.21.15.86:0/2187759461 learned_addr learned my addr 172.21.15.86:0/2187759461 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a496d640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7a0101b50 0x7fa7a010e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a516e640 1 -- 172.21.15.86:0/2187759461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7a0101b50 msgr2=0x7fa7a010e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a516e640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7a0101b50 0x7fa7a010e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a516e640 1 -- 172.21.15.86:0/2187759461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790005040 con 0x7fa7a01011b0 2024-06-06T08:28:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a516e640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 0x7fa7a010e080 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa790003080 tx=0x7fa790009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa795ffb640 1 -- 172.21.15.86:0/2187759461 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa790038070 con 0x7fa7a01011b0 2024-06-06T08:28:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa795ffb640 1 -- 172.21.15.86:0/2187759461 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa790008ee0 con 0x7fa7a01011b0 2024-06-06T08:28:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a01170f0 con 0x7fa7a01011b0 2024-06-06T08:28:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa795ffb640 1 -- 172.21.15.86:0/2187759461 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa79003d870 con 0x7fa7a01011b0 2024-06-06T08:28:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.530+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a0117560 con 0x7fa7a01011b0 2024-06-06T08:28:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.531+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a00fc920 con 0x7fa7a01011b0 2024-06-06T08:28:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.532+0000 7fa795ffb640 1 -- 172.21.15.86:0/2187759461 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa790030020 con 0x7fa7a01011b0 2024-06-06T08:28:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.532+0000 7fa795ffb640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa78c065430 0x7fa78c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.532+0000 7fa7a496d640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa78c065430 0x7fa78c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.532+0000 7fa795ffb640 1 -- 172.21.15.86:0/2187759461 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa7900a2240 con 0x7fa7a01011b0 2024-06-06T08:28:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.533+0000 7fa7a496d640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa78c065430 0x7fa78c0678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00661b0 tx=0x7fa79c008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.536+0000 7fa795ffb640 1 -- 172.21.15.86:0/2187759461 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa790070770 con 0x7fa7a01011b0 2024-06-06T08:28:21.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.734+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa7a01083a0 con 0x7fa7a01011b0 2024-06-06T08:28:21.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.734+0000 7fa795ffb640 1 -- 172.21.15.86:0/2187759461 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa790073d90 con 0x7fa7a01011b0 2024-06-06T08:28:21.734 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:21.734 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:21.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa78c065430 msgr2=0x7fa78c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:21.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa78c065430 0x7fa78c0678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00661b0 tx=0x7fa79c008040 comp rx=0 tx=0).stop 2024-06-06T08:28:21.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 msgr2=0x7fa7a010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:21.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 0x7fa7a010e080 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa790003080 tx=0x7fa790009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:21.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 shutdown_connections 2024-06-06T08:28:21.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7a0101b50 0x7fa7a010e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:21.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa78c065430 0x7fa78c0678e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:21.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 --2- 172.21.15.86:0/2187759461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a01011b0 0x7fa7a010e080 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:21.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 >> 172.21.15.86:0/2187759461 conn(0x7fa7a00f6d60 msgr2=0x7fa7a00f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:21.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 shutdown_connections 2024-06-06T08:28:21.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:21.736+0000 7fa7a73d2640 1 -- 172.21.15.86:0/2187759461 wait complete. 2024-06-06T08:28:22.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:22 smithi154 ceph-mon[35428]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:22.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:22 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2187759461' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:22.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:22 smithi086 ceph-mon[29114]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:22.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:22 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2187759461' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:24.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:24 smithi154 ceph-mon[35428]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:24.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:24 smithi086 ceph-mon[29114]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:25.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:25.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:25 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:28:25.518 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:25.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:25 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:28:26.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:26 smithi086 ceph-mon[29114]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:26.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:26 smithi154 ceph-mon[35428]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:27.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.015+0000 7fdd6780f640 1 -- 172.21.15.86:0/2415634070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd60101b50 msgr2=0x7fdd600fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.015+0000 7fdd6780f640 1 --2- 172.21.15.86:0/2415634070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd60101b50 0x7fdd600fb1e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdd500048d0 tx=0x7fdd50009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.015+0000 7fdd6780f640 1 -- 172.21.15.86:0/2415634070 shutdown_connections 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.015+0000 7fdd6780f640 1 --2- 172.21.15.86:0/2415634070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd60101b50 0x7fdd600fb1e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.015+0000 7fdd6780f640 1 --2- 172.21.15.86:0/2415634070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd601011b0 0x7fdd60101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.015+0000 7fdd6780f640 1 -- 172.21.15.86:0/2415634070 >> 172.21.15.86:0/2415634070 conn(0x7fdd600f6d60 msgr2=0x7fdd600f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.015+0000 7fdd6780f640 1 -- 172.21.15.86:0/2415634070 shutdown_connections 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.015+0000 7fdd6780f640 1 -- 172.21.15.86:0/2415634070 wait complete. 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd6780f640 1 Processor -- start 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd6780f640 1 -- start start 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd6780f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd601011b0 0x7fdd6010dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd6780f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd60101b50 0x7fdd6010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd6780f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6010eb40 con 0x7fdd601011b0 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd6780f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6010ecb0 con 0x7fdd60101b50 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd655ab640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd601011b0 0x7fdd6010dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd655ab640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd601011b0 0x7fdd6010dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48888/0 (socket says 172.21.15.86:48888) 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd655ab640 1 -- 172.21.15.86:0/3223015369 learned_addr learned my addr 172.21.15.86:0/3223015369 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.016+0000 7fdd64daa640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd60101b50 0x7fdd6010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.017+0000 7fdd655ab640 1 -- 172.21.15.86:0/3223015369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd60101b50 msgr2=0x7fdd6010e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.017+0000 7fdd655ab640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd60101b50 0x7fdd6010e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.017+0000 7fdd655ab640 1 -- 172.21.15.86:0/3223015369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd50005040 con 0x7fdd601011b0 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.017+0000 7fdd655ab640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd601011b0 0x7fdd6010dfb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00c830 tx=0x7fdd5c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.017+0000 7fdd55ffb640 1 -- 172.21.15.86:0/3223015369 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd5c00a7d0 con 0x7fdd601011b0 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.017+0000 7fdd55ffb640 1 -- 172.21.15.86:0/3223015369 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd5c00ae00 con 0x7fdd601011b0 2024-06-06T08:28:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.017+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd60117020 con 0x7fdd601011b0 2024-06-06T08:28:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.017+0000 7fdd55ffb640 1 -- 172.21.15.86:0/3223015369 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd5c0113a0 con 0x7fdd601011b0 2024-06-06T08:28:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.017+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd60117530 con 0x7fdd601011b0 2024-06-06T08:28:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.018+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd600fc920 con 0x7fdd601011b0 2024-06-06T08:28:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.019+0000 7fdd55ffb640 1 -- 172.21.15.86:0/3223015369 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fdd5c012400 con 0x7fdd601011b0 2024-06-06T08:28:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.019+0000 7fdd55ffb640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd4c065430 0x7fdd4c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.019+0000 7fdd55ffb640 1 -- 172.21.15.86:0/3223015369 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdd5c082580 con 0x7fdd601011b0 2024-06-06T08:28:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.019+0000 7fdd64daa640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd4c065430 0x7fdd4c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.019+0000 7fdd64daa640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd4c065430 0x7fdd4c0678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009f80 tx=0x7fdd50035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.023+0000 7fdd55ffb640 1 -- 172.21.15.86:0/3223015369 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdd5c050ab0 con 0x7fdd601011b0 2024-06-06T08:28:27.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.223+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdd600fa9a0 con 0x7fdd601011b0 2024-06-06T08:28:27.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.223+0000 7fdd55ffb640 1 -- 172.21.15.86:0/3223015369 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdd5c0540d0 con 0x7fdd601011b0 2024-06-06T08:28:27.223 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:27.223 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:27.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd4c065430 msgr2=0x7fdd4c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:27.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd4c065430 0x7fdd4c0678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009f80 tx=0x7fdd50035040 comp rx=0 tx=0).stop 2024-06-06T08:28:27.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd601011b0 msgr2=0x7fdd6010dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:27.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd601011b0 0x7fdd6010dfb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00c830 tx=0x7fdd5c004430 comp rx=0 tx=0).stop 2024-06-06T08:28:27.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 shutdown_connections 2024-06-06T08:28:27.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd60101b50 0x7fdd6010e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:27.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd4c065430 0x7fdd4c0678e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:27.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 --2- 172.21.15.86:0/3223015369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd601011b0 0x7fdd6010dfb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:27.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 >> 172.21.15.86:0/3223015369 conn(0x7fdd600f6d60 msgr2=0x7fdd600f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:27.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 shutdown_connections 2024-06-06T08:28:27.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:27.225+0000 7fdd6780f640 1 -- 172.21.15.86:0/3223015369 wait complete. 2024-06-06T08:28:28.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:28 smithi154 ceph-mon[35428]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:28.418 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:28 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3223015369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:28.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:28 smithi086 ceph-mon[29114]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:28.624 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:28 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3223015369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:30 smithi154 ceph-mon[35428]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:30.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:30 smithi086 ceph-mon[29114]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:30.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:31.046 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:32.408 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:32 smithi086 ceph-mon[29114]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:32.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:32 smithi154 ceph-mon[35428]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:32.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 -- 172.21.15.86:0/834358192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 msgr2=0x7f906c102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 --2- 172.21.15.86:0/834358192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 0x7f906c102690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f905c0048d0 tx=0x7f905c009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 -- 172.21.15.86:0/834358192 shutdown_connections 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 --2- 172.21.15.86:0/834358192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c102c60 0x7f906c0fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 --2- 172.21.15.86:0/834358192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 0x7f906c102690 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 -- 172.21.15.86:0/834358192 >> 172.21.15.86:0/834358192 conn(0x7f906c0f7e50 msgr2=0x7f906c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 -- 172.21.15.86:0/834358192 shutdown_connections 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 -- 172.21.15.86:0/834358192 wait complete. 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 Processor -- start 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.486+0000 7f90747ab640 1 -- start start 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f90747ab640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 0x7f906c1133d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f90747ab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c102c60 0x7f906c113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f90747ab640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c113fa0 con 0x7f906c1022c0 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f90747ab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c10d350 con 0x7f906c102c60 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f9072547640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 0x7f906c1133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f9072547640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 0x7f906c1133d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48906/0 (socket says 172.21.15.86:48906) 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f9072547640 1 -- 172.21.15.86:0/1487839914 learned_addr learned my addr 172.21.15.86:0/1487839914 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f9071d46640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c102c60 0x7f906c113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f9072547640 1 -- 172.21.15.86:0/1487839914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c102c60 msgr2=0x7f906c113910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f9072547640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c102c60 0x7f906c113910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f9072547640 1 -- 172.21.15.86:0/1487839914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c005040 con 0x7f906c1022c0 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f9072547640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 0x7f906c1133d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f905c003080 tx=0x7f905c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f906affd640 1 -- 172.21.15.86:0/1487839914 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f905c038070 con 0x7f906c1022c0 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f906affd640 1 -- 172.21.15.86:0/1487839914 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f905c008ee0 con 0x7f906c1022c0 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f906c10d5d0 con 0x7f906c1022c0 2024-06-06T08:28:32.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f906affd640 1 -- 172.21.15.86:0/1487839914 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f905c03d870 con 0x7f906c1022c0 2024-06-06T08:28:32.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.487+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906c10da90 con 0x7f906c1022c0 2024-06-06T08:28:32.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.488+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906c0fda30 con 0x7f906c1022c0 2024-06-06T08:28:32.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.489+0000 7f906affd640 1 -- 172.21.15.86:0/1487839914 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f905c030020 con 0x7f906c1022c0 2024-06-06T08:28:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.489+0000 7f906affd640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9054065430 0x7f90540678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.489+0000 7f906affd640 1 -- 172.21.15.86:0/1487839914 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f905c0a2240 con 0x7f906c1022c0 2024-06-06T08:28:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.489+0000 7f9071d46640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9054065430 0x7f90540678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.490+0000 7f9071d46640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9054065430 0x7f90540678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f906c10ec90 tx=0x7f9060009590 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.493+0000 7f906affd640 1 -- 172.21.15.86:0/1487839914 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f905c070770 con 0x7f906c1022c0 2024-06-06T08:28:32.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.692+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f906c0fbab0 con 0x7f906c1022c0 2024-06-06T08:28:32.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.692+0000 7f906affd640 1 -- 172.21.15.86:0/1487839914 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f905c073d90 con 0x7f906c1022c0 2024-06-06T08:28:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:32.692 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9054065430 msgr2=0x7f90540678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9054065430 0x7f90540678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f906c10ec90 tx=0x7f9060009590 comp rx=0 tx=0).stop 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 msgr2=0x7f906c1133d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 0x7f906c1133d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f905c003080 tx=0x7f905c009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 shutdown_connections 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c102c60 0x7f906c113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9054065430 0x7f90540678e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 --2- 172.21.15.86:0/1487839914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f906c1022c0 0x7f906c1133d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 >> 172.21.15.86:0/1487839914 conn(0x7f906c0f7e50 msgr2=0x7f906c0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 shutdown_connections 2024-06-06T08:28:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:32.694+0000 7f90747ab640 1 -- 172.21.15.86:0/1487839914 wait complete. 2024-06-06T08:28:33.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:33 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1487839914' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:33.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:33 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1487839914' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:34.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:34 smithi154 ceph-mon[35428]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:34.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:34 smithi086 ceph-mon[29114]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:36.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:36.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:36 smithi154 ceph-mon[35428]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:36.580 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:36.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:36 smithi086 ceph-mon[29114]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:37.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.929+0000 7fc14afa6640 1 -- 172.21.15.86:0/1287971495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 msgr2=0x7fc144102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:37.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.929+0000 7fc14afa6640 1 --2- 172.21.15.86:0/1287971495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 0x7fc144102750 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc1400048d0 tx=0x7fc140009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:37.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.929+0000 7fc14afa6640 1 -- 172.21.15.86:0/1287971495 shutdown_connections 2024-06-06T08:28:37.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.929+0000 7fc14afa6640 1 --2- 172.21.15.86:0/1287971495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144102d20 0x7fc1440faa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:37.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.929+0000 7fc14afa6640 1 --2- 172.21.15.86:0/1287971495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 0x7fc144102750 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:37.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.930+0000 7fc14afa6640 1 -- 172.21.15.86:0/1287971495 >> 172.21.15.86:0/1287971495 conn(0x7fc144070ab0 msgr2=0x7fc144070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:37.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.930+0000 7fc14afa6640 1 -- 172.21.15.86:0/1287971495 shutdown_connections 2024-06-06T08:28:37.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.930+0000 7fc14afa6640 1 -- 172.21.15.86:0/1287971495 wait complete. 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.930+0000 7fc14afa6640 1 Processor -- start 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.930+0000 7fc14afa6640 1 -- start start 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.930+0000 7fc14afa6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 0x7fc1441130b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc14afa6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144102d20 0x7fc1441135f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc14afa6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc144113bf0 con 0x7fc144102380 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc14afa6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14410d030 con 0x7fc144102d20 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc148d42640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 0x7fc1441130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc148d42640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 0x7fc1441130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35310/0 (socket says 172.21.15.86:35310) 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc148d42640 1 -- 172.21.15.86:0/1887679192 learned_addr learned my addr 172.21.15.86:0/1887679192 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc13bfff640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144102d20 0x7fc1441135f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:37.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc148d42640 1 -- 172.21.15.86:0/1887679192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144102d20 msgr2=0x7fc1441135f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc148d42640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144102d20 0x7fc1441135f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc148d42640 1 -- 172.21.15.86:0/1887679192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc140005040 con 0x7fc144102380 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc148d42640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 0x7fc1441130b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc140003080 tx=0x7fc140009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc1397fa640 1 -- 172.21.15.86:0/1887679192 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc140038070 con 0x7fc144102380 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc1397fa640 1 -- 172.21.15.86:0/1887679192 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc140008ee0 con 0x7fc144102380 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc1397fa640 1 -- 172.21.15.86:0/1887679192 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc14003d870 con 0x7fc144102380 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14410d2b0 con 0x7fc144102380 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.931+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14410d770 con 0x7fc144102380 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.932+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc144065520 con 0x7fc144102380 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.933+0000 7fc1397fa640 1 -- 172.21.15.86:0/1887679192 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fc140030020 con 0x7fc144102380 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.933+0000 7fc1397fa640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc12c065430 0x7fc12c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.933+0000 7fc1397fa640 1 -- 172.21.15.86:0/1887679192 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc1400a2150 con 0x7fc144102380 2024-06-06T08:28:37.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.933+0000 7fc13bfff640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc12c065430 0x7fc12c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:37.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.933+0000 7fc13bfff640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc12c065430 0x7fc12c0678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc14410e970 tx=0x7fc134009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:37.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:37.938+0000 7fc1397fa640 1 -- 172.21.15.86:0/1887679192 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc140070720 con 0x7fc144102380 2024-06-06T08:28:38.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.137+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc14404b480 con 0x7fc144102380 2024-06-06T08:28:38.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.138+0000 7fc1397fa640 1 -- 172.21.15.86:0/1887679192 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc140073d40 con 0x7fc144102380 2024-06-06T08:28:38.138 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:38.138 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:38.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.139+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc12c065430 msgr2=0x7fc12c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.139+0000 7fc14afa6640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc12c065430 0x7fc12c0678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc14410e970 tx=0x7fc134009040 comp rx=0 tx=0).stop 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.139+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 msgr2=0x7fc1441130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.139+0000 7fc14afa6640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 0x7fc1441130b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc140003080 tx=0x7fc140009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.139+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 shutdown_connections 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.139+0000 7fc14afa6640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144102d20 0x7fc1441135f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.139+0000 7fc14afa6640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc12c065430 0x7fc12c0678e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.139+0000 7fc14afa6640 1 --2- 172.21.15.86:0/1887679192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc144102380 0x7fc1441130b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.139+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 >> 172.21.15.86:0/1887679192 conn(0x7fc144070ab0 msgr2=0x7fc1440fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.140+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 shutdown_connections 2024-06-06T08:28:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:38.140+0000 7fc14afa6640 1 -- 172.21.15.86:0/1887679192 wait complete. 2024-06-06T08:28:38.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:38 smithi086 ceph-mon[29114]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:38.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:38 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1887679192' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:38.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:38 smithi154 ceph-mon[35428]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:38.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:38 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1887679192' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:40.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:40 smithi154 ceph-mon[35428]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:40.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:40 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:28:40.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:40 smithi086 ceph-mon[29114]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:40.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:28:41.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:41.913 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:42.276 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:42 smithi086 ceph-mon[29114]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:42.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:42 smithi154 ceph-mon[35428]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:43.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.287+0000 7fc11c718640 1 -- 172.21.15.86:0/742683261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc114102c40 msgr2=0x7fc1140fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:43.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.287+0000 7fc11c718640 1 --2- 172.21.15.86:0/742683261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc114102c40 0x7fc1140fc2d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc1040048d0 tx=0x7fc104009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:43.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.287+0000 7fc11c718640 1 -- 172.21.15.86:0/742683261 shutdown_connections 2024-06-06T08:28:43.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.287+0000 7fc11c718640 1 --2- 172.21.15.86:0/742683261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc114102c40 0x7fc1140fc2d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:43.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.287+0000 7fc11c718640 1 --2- 172.21.15.86:0/742683261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1141022a0 0x7fc114102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:43.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.287+0000 7fc11c718640 1 -- 172.21.15.86:0/742683261 >> 172.21.15.86:0/742683261 conn(0x7fc1140f7e50 msgr2=0x7fc1140fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:43.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.287+0000 7fc11c718640 1 -- 172.21.15.86:0/742683261 shutdown_connections 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.287+0000 7fc11c718640 1 -- 172.21.15.86:0/742683261 wait complete. 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11c718640 1 Processor -- start 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11c718640 1 -- start start 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11c718640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1141022a0 0x7fc114113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11c718640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc114102c40 0x7fc114113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11c718640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc114113b60 con 0x7fc1141022a0 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11c718640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11410cfa0 con 0x7fc114102c40 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11a4b4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1141022a0 0x7fc114113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11a4b4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1141022a0 0x7fc114113020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35318/0 (socket says 172.21.15.86:35318) 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11a4b4640 1 -- 172.21.15.86:0/2492605181 learned_addr learned my addr 172.21.15.86:0/2492605181 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc119cb3640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc114102c40 0x7fc114113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:43.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.288+0000 7fc11a4b4640 1 -- 172.21.15.86:0/2492605181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc114102c40 msgr2=0x7fc114113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.289+0000 7fc11a4b4640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc114102c40 0x7fc114113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.289+0000 7fc11a4b4640 1 -- 172.21.15.86:0/2492605181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc104005040 con 0x7fc1141022a0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.289+0000 7fc11a4b4640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1141022a0 0x7fc114113020 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc110002a50 tx=0x7fc110004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.289+0000 7fc10affd640 1 -- 172.21.15.86:0/2492605181 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc110007c30 con 0x7fc1141022a0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.289+0000 7fc10affd640 1 -- 172.21.15.86:0/2492605181 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc11000f040 con 0x7fc1141022a0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.289+0000 7fc10affd640 1 -- 172.21.15.86:0/2492605181 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc11000dad0 con 0x7fc1141022a0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.289+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11410d280 con 0x7fc1141022a0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.289+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11410d7f0 con 0x7fc1141022a0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.290+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1140fda10 con 0x7fc1141022a0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.290+0000 7fc10affd640 1 -- 172.21.15.86:0/2492605181 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fc11000dc70 con 0x7fc1141022a0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.291+0000 7fc10affd640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc100065430 0x7fc1000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.291+0000 7fc10affd640 1 -- 172.21.15.86:0/2492605181 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc110081bb0 con 0x7fc1141022a0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.291+0000 7fc119cb3640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc100065430 0x7fc1000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:43.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.291+0000 7fc119cb3640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc100065430 0x7fc1000678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc104032430 tx=0x7fc104035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:43.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.295+0000 7fc10affd640 1 -- 172.21.15.86:0/2492605181 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc11004cdf0 con 0x7fc1141022a0 2024-06-06T08:28:43.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.494+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc114109400 con 0x7fc1141022a0 2024-06-06T08:28:43.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.495+0000 7fc10affd640 1 -- 172.21.15.86:0/2492605181 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc110088080 con 0x7fc1141022a0 2024-06-06T08:28:43.494 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:43.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc100065430 msgr2=0x7fc1000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:43.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc100065430 0x7fc1000678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc104032430 tx=0x7fc104035040 comp rx=0 tx=0).stop 2024-06-06T08:28:43.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1141022a0 msgr2=0x7fc114113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:43.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1141022a0 0x7fc114113020 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc110002a50 tx=0x7fc110004600 comp rx=0 tx=0).stop 2024-06-06T08:28:43.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 shutdown_connections 2024-06-06T08:28:43.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc114102c40 0x7fc114113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:43.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fc100065430 0x7fc1000678e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:43.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 --2- 172.21.15.86:0/2492605181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1141022a0 0x7fc114113020 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:43.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 >> 172.21.15.86:0/2492605181 conn(0x7fc1140f7e50 msgr2=0x7fc1140f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:43.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 shutdown_connections 2024-06-06T08:28:43.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:43.497+0000 7fc11c718640 1 -- 172.21.15.86:0/2492605181 wait complete. 2024-06-06T08:28:44.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:44 smithi086 ceph-mon[29114]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:44.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:44 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2492605181' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:44.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:44 smithi154 ceph-mon[35428]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:44.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:44 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2492605181' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:46.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:46 smithi086 ceph-mon[29114]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:46.602 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:46 smithi154 ceph-mon[35428]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:47.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:48.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:48 smithi086 ceph-mon[29114]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:48.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:28:48.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:28:48.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:28:48.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:28:48.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:48 smithi154 ceph-mon[35428]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:48.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:28:48.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:28:48.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:28:48.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:28:48.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.774+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/3510869125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 msgr2=0x7f7e9c0fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:48.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.774+0000 7f7ea0d7e640 1 --2- 172.21.15.86:0/3510869125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 0x7f7e9c0fae30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7e840048d0 tx=0x7f7e84009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:48.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.774+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/3510869125 shutdown_connections 2024-06-06T08:28:48.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.774+0000 7f7ea0d7e640 1 --2- 172.21.15.86:0/3510869125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 0x7f7e9c0fae30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:48.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.774+0000 7f7ea0d7e640 1 --2- 172.21.15.86:0/3510869125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e9c102370 0x7f7e9c102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:48.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.774+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/3510869125 >> 172.21.15.86:0/3510869125 conn(0x7f7e9c0709e0 msgr2=0x7f7e9c070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:48.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.774+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/3510869125 shutdown_connections 2024-06-06T08:28:48.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.774+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/3510869125 wait complete. 2024-06-06T08:28:48.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7ea0d7e640 1 Processor -- start 2024-06-06T08:28:48.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7ea0d7e640 1 -- start start 2024-06-06T08:28:48.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7ea0d7e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 0x7f7e9c113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7ea0d7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e9c102370 0x7f7e9c113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7ea0d7e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c113b40 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7ea0d7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c10cf80 con 0x7f7e9c102370 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7e9a59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 0x7f7e9c113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7e9a59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 0x7f7e9c113000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39732/0 (socket says 172.21.15.86:39732) 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7e9a59c640 1 -- 172.21.15.86:0/202867532 learned_addr learned my addr 172.21.15.86:0/202867532 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7e99d9b640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e9c102370 0x7f7e9c113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7e9a59c640 1 -- 172.21.15.86:0/202867532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e9c102370 msgr2=0x7f7e9c113540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7e9a59c640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e9c102370 0x7f7e9c113540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.775+0000 7f7e9a59c640 1 -- 172.21.15.86:0/202867532 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84005040 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.776+0000 7f7e9a59c640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 0x7f7e9c113000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7e9400cae0 tx=0x7f7e94007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.776+0000 7f7e92ffd640 1 -- 172.21.15.86:0/202867532 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e9400aa80 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.776+0000 7f7e92ffd640 1 -- 172.21.15.86:0/202867532 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e9400abe0 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.776+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9c10d260 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.776+0000 7f7e92ffd640 1 -- 172.21.15.86:0/202867532 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e940116b0 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.776+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9c10d7d0 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.777+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9c065570 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.777+0000 7f7e92ffd640 1 -- 172.21.15.86:0/202867532 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f7e9400ad50 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.778+0000 7f7e92ffd640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e88065430 0x7f7e880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.778+0000 7f7e92ffd640 1 -- 172.21.15.86:0/202867532 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7e94082ca0 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.778+0000 7f7e99d9b640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e88065430 0x7f7e880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:48.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.778+0000 7f7e99d9b640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e88065430 0x7f7e880678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009f80 tx=0x7f7e84035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:48.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.782+0000 7f7e92ffd640 1 -- 172.21.15.86:0/202867532 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7e94087070 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.978+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7e9c109540 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.978+0000 7f7e92ffd640 1 -- 172.21.15.86:0/202867532 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7e94051150 con 0x7f7e9c0fa9c0 2024-06-06T08:28:48.978 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:48.978 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.980+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e88065430 msgr2=0x7f7e880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.980+0000 7f7ea0d7e640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e88065430 0x7f7e880678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009f80 tx=0x7f7e84035040 comp rx=0 tx=0).stop 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.980+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 msgr2=0x7f7e9c113000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.980+0000 7f7ea0d7e640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 0x7f7e9c113000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7e9400cae0 tx=0x7f7e94007e50 comp rx=0 tx=0).stop 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.981+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 shutdown_connections 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.981+0000 7f7ea0d7e640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e9c102370 0x7f7e9c113540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.981+0000 7f7ea0d7e640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e88065430 0x7f7e880678e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.981+0000 7f7ea0d7e640 1 --2- 172.21.15.86:0/202867532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e9c0fa9c0 0x7f7e9c113000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.981+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 >> 172.21.15.86:0/202867532 conn(0x7f7e9c0709e0 msgr2=0x7f7e9c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.981+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 shutdown_connections 2024-06-06T08:28:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:48.981+0000 7f7ea0d7e640 1 -- 172.21.15.86:0/202867532 wait complete. 2024-06-06T08:28:49.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:49 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/202867532' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:49.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:49 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/202867532' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:50.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:50 smithi086 ceph-mon[29114]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:50.666 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:50 smithi154 ceph-mon[35428]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:52.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:52 smithi086 ceph-mon[29114]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:52.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:52.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:52 smithi154 ceph-mon[35428]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:52.865 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:54.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 -- 172.21.15.86:0/2848399885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 msgr2=0x7f5f900fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:54.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 --2- 172.21.15.86:0/2848399885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 0x7f5f900fc2f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5f780048d0 tx=0x7f5f78009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:54.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 -- 172.21.15.86:0/2848399885 shutdown_connections 2024-06-06T08:28:54.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 --2- 172.21.15.86:0/2848399885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 0x7f5f900fc2f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:54.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 --2- 172.21.15.86:0/2848399885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f901022c0 0x7f5f90102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:54.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 -- 172.21.15.86:0/2848399885 >> 172.21.15.86:0/2848399885 conn(0x7f5f900f7e50 msgr2=0x7f5f900fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:54.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 -- 172.21.15.86:0/2848399885 shutdown_connections 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 -- 172.21.15.86:0/2848399885 wait complete. 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 Processor -- start 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.142+0000 7f5f96886640 1 -- start start 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f96886640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f901022c0 0x7f5f901133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f96886640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 0x7f5f90113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f96886640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f90113f90 con 0x7f5f90102c60 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f96886640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9010d340 con 0x7f5f901022c0 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f87fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 0x7f5f90113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f87fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 0x7f5f90113900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39748/0 (socket says 172.21.15.86:39748) 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f87fff640 1 -- 172.21.15.86:0/1146889521 learned_addr learned my addr 172.21.15.86:0/1146889521 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f8ffff640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f901022c0 0x7f5f901133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f87fff640 1 -- 172.21.15.86:0/1146889521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f901022c0 msgr2=0x7f5f901133c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f87fff640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f901022c0 0x7f5f901133c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f87fff640 1 -- 172.21.15.86:0/1146889521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f78005040 con 0x7f5f90102c60 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f8ffff640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f901022c0 0x7f5f901133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f87fff640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 0x7f5f90113900 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5f78003080 tx=0x7f5f78009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f8d7fa640 1 -- 172.21.15.86:0/1146889521 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f78038070 con 0x7f5f90102c60 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9010d5c0 con 0x7f5f90102c60 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f8d7fa640 1 -- 172.21.15.86:0/1146889521 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f78008ee0 con 0x7f5f90102c60 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f8d7fa640 1 -- 172.21.15.86:0/1146889521 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f7803d870 con 0x7f5f90102c60 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.143+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9010da80 con 0x7f5f90102c60 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.145+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f900fda30 con 0x7f5f90102c60 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.145+0000 7f5f8d7fa640 1 -- 172.21.15.86:0/1146889521 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f5f78030020 con 0x7f5f90102c60 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.145+0000 7f5f8d7fa640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f5f7c065430 0x7f5f7c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.145+0000 7f5f8d7fa640 1 -- 172.21.15.86:0/1146889521 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5f780a2240 con 0x7f5f90102c60 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.145+0000 7f5f8ffff640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f5f7c065430 0x7f5f7c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.146+0000 7f5f8ffff640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f5f7c065430 0x7f5f7c0678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007930 tx=0x7f5f80008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:28:54.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.149+0000 7f5f8d7fa640 1 -- 172.21.15.86:0/1146889521 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5f78070770 con 0x7f5f90102c60 2024-06-06T08:28:54.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.349+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5f901094e0 con 0x7f5f90102c60 2024-06-06T08:28:54.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.349+0000 7f5f8d7fa640 1 -- 172.21.15.86:0/1146889521 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5f78073d90 con 0x7f5f90102c60 2024-06-06T08:28:54.349 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:28:54.349 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:28:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f5f7c065430 msgr2=0x7f5f7c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f5f7c065430 0x7f5f7c0678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5f80007930 tx=0x7f5f80008040 comp rx=0 tx=0).stop 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 msgr2=0x7f5f90113900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 0x7f5f90113900 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5f78003080 tx=0x7f5f78009da0 comp rx=0 tx=0).stop 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 shutdown_connections 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f5f7c065430 0x7f5f7c0678e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f90102c60 0x7f5f90113900 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 --2- 172.21.15.86:0/1146889521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f901022c0 0x7f5f901133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 >> 172.21.15.86:0/1146889521 conn(0x7f5f900f7e50 msgr2=0x7f5f900f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 shutdown_connections 2024-06-06T08:28:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:54.351+0000 7f5f96886640 1 -- 172.21.15.86:0/1146889521 wait complete. 2024-06-06T08:28:54.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:54 smithi086 ceph-mon[29114]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:54.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:54 smithi154 ceph-mon[35428]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:55.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:55 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1146889521' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:55.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:55 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:28:55.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:55 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1146889521' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:28:55.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:55 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:28:56.251 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:56 smithi086 ceph-mon[29114]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:56.601 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:56 smithi154 ceph-mon[35428]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:57.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:28:58.254 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:28:58.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:28:58 smithi086 ceph-mon[29114]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:58.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:28:58 smithi154 ceph-mon[35428]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:28:59.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 -- 172.21.15.86:0/1078378194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef301011a0 msgr2=0x7fef30101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:00.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 --2- 172.21.15.86:0/1078378194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef301011a0 0x7fef30101570 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fef200048d0 tx=0x7fef20009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:00.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 -- 172.21.15.86:0/1078378194 shutdown_connections 2024-06-06T08:29:00.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 --2- 172.21.15.86:0/1078378194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef30101b40 0x7fef300fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:00.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 --2- 172.21.15.86:0/1078378194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef301011a0 0x7fef30101570 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:00.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 -- 172.21.15.86:0/1078378194 >> 172.21.15.86:0/1078378194 conn(0x7fef300f6d90 msgr2=0x7fef300f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:00.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 -- 172.21.15.86:0/1078378194 shutdown_connections 2024-06-06T08:29:00.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 -- 172.21.15.86:0/1078378194 wait complete. 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 Processor -- start 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 -- start start 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.885+0000 7fef35069640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef301011a0 0x7fef30106060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef35069640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef30101b40 0x7fef301065a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef35069640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef30106ae0 con 0x7fef30101b40 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef35069640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef30106c50 con 0x7fef301011a0 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef27fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef30101b40 0x7fef301065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef27fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef30101b40 0x7fef301065a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:41442/0 (socket says 172.21.15.86:41442) 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef27fff640 1 -- 172.21.15.86:0/363815568 learned_addr learned my addr 172.21.15.86:0/363815568 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef2ed9d640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef301011a0 0x7fef30106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:00.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef27fff640 1 -- 172.21.15.86:0/363815568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef301011a0 msgr2=0x7fef30106060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef27fff640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef301011a0 0x7fef30106060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef27fff640 1 -- 172.21.15.86:0/363815568 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef20005040 con 0x7fef30101b40 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef2ed9d640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef301011a0 0x7fef30106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef27fff640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef30101b40 0x7fef301065a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fef1800c830 tx=0x7fef18004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef277fe640 1 -- 172.21.15.86:0/363815568 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef1800a850 con 0x7fef30101b40 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef277fe640 1 -- 172.21.15.86:0/363815568 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef1800ae80 con 0x7fef30101b40 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef30106ee0 con 0x7fef30101b40 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef277fe640 1 -- 172.21.15.86:0/363815568 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef18012ea0 con 0x7fef30101b40 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.886+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef301196e0 con 0x7fef30101b40 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.887+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef300fc910 con 0x7fef30101b40 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.888+0000 7fef277fe640 1 -- 172.21.15.86:0/363815568 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fef18012400 con 0x7fef30101b40 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.888+0000 7fef277fe640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fef1c065430 0x7fef1c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.888+0000 7fef2ed9d640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fef1c065430 0x7fef1c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.888+0000 7fef277fe640 1 -- 172.21.15.86:0/363815568 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fef18082730 con 0x7fef30101b40 2024-06-06T08:29:00.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.888+0000 7fef2ed9d640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fef1c065430 0x7fef1c0678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fef20009f80 tx=0x7fef20035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:00.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:28:59.892+0000 7fef277fe640 1 -- 172.21.15.86:0/363815568 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fef18050c60 con 0x7fef30101b40 2024-06-06T08:29:00.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.088+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fef30107750 con 0x7fef30101b40 2024-06-06T08:29:00.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.088+0000 7fef277fe640 1 -- 172.21.15.86:0/363815568 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fef180126b0 con 0x7fef30101b40 2024-06-06T08:29:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:00.105 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fef1c065430 msgr2=0x7fef1c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fef1c065430 0x7fef1c0678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fef20009f80 tx=0x7fef20035040 comp rx=0 tx=0).stop 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef30101b40 msgr2=0x7fef301065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef30101b40 0x7fef301065a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fef1800c830 tx=0x7fef18004430 comp rx=0 tx=0).stop 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 shutdown_connections 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fef1c065430 0x7fef1c0678e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef30101b40 0x7fef301065a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 --2- 172.21.15.86:0/363815568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef301011a0 0x7fef30106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 >> 172.21.15.86:0/363815568 conn(0x7fef300f6d90 msgr2=0x7fef300f7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:00.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 shutdown_connections 2024-06-06T08:29:00.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:00.090+0000 7fef35069640 1 -- 172.21.15.86:0/363815568 wait complete. 2024-06-06T08:29:00.387 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:00 smithi086 ceph-mon[29114]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:00.387 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:00 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/363815568' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:00.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:00 smithi154 ceph-mon[35428]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:00.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:00 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/363815568' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:02.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:02 smithi086 ceph-mon[29114]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:02.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:02 smithi154 ceph-mon[35428]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:03.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:03.901 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:04.290 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:04 smithi086 ceph-mon[29114]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:04.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:04 smithi154 ceph-mon[35428]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.456+0000 7fbf3d057640 1 -- 172.21.15.86:0/3045648879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 msgr2=0x7fbf38101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.456+0000 7fbf3d057640 1 --2- 172.21.15.86:0/3045648879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 0x7fbf38101580 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbf280048d0 tx=0x7fbf28009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.456+0000 7fbf3d057640 1 -- 172.21.15.86:0/3045648879 shutdown_connections 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.456+0000 7fbf3d057640 1 --2- 172.21.15.86:0/3045648879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf38101b50 0x7fbf380fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.456+0000 7fbf3d057640 1 --2- 172.21.15.86:0/3045648879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 0x7fbf38101580 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.456+0000 7fbf3d057640 1 -- 172.21.15.86:0/3045648879 >> 172.21.15.86:0/3045648879 conn(0x7fbf380f6da0 msgr2=0x7fbf380f91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.456+0000 7fbf3d057640 1 -- 172.21.15.86:0/3045648879 shutdown_connections 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.456+0000 7fbf3d057640 1 -- 172.21.15.86:0/3045648879 wait complete. 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.456+0000 7fbf3d057640 1 Processor -- start 2024-06-06T08:29:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf3d057640 1 -- start start 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf3d057640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 0x7fbf3810e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf3d057640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf38101b50 0x7fbf3810e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf3d057640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3810ec30 con 0x7fbf381011b0 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf3d057640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3810eda0 con 0x7fbf38101b50 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf36d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 0x7fbf3810e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf36d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 0x7fbf3810e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:41460/0 (socket says 172.21.15.86:41460) 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf36d9d640 1 -- 172.21.15.86:0/377220396 learned_addr learned my addr 172.21.15.86:0/377220396 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf2ffff640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf38101b50 0x7fbf3810e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf36d9d640 1 -- 172.21.15.86:0/377220396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf38101b50 msgr2=0x7fbf3810e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf36d9d640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf38101b50 0x7fbf3810e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf36d9d640 1 -- 172.21.15.86:0/377220396 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf28005040 con 0x7fbf381011b0 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf2ffff640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf38101b50 0x7fbf3810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:29:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf36d9d640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 0x7fbf3810e010 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbf28003080 tx=0x7fbf28009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:05.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.457+0000 7fbf2f7fe640 1 -- 172.21.15.86:0/377220396 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf28038070 con 0x7fbf381011b0 2024-06-06T08:29:05.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.458+0000 7fbf2f7fe640 1 -- 172.21.15.86:0/377220396 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf28008ee0 con 0x7fbf381011b0 2024-06-06T08:29:05.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.458+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf38117080 con 0x7fbf381011b0 2024-06-06T08:29:05.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.458+0000 7fbf2f7fe640 1 -- 172.21.15.86:0/377220396 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf2803d870 con 0x7fbf381011b0 2024-06-06T08:29:05.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.458+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf381174f0 con 0x7fbf381011b0 2024-06-06T08:29:05.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.458+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf380fc920 con 0x7fbf381011b0 2024-06-06T08:29:05.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.459+0000 7fbf2f7fe640 1 -- 172.21.15.86:0/377220396 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fbf28030020 con 0x7fbf381011b0 2024-06-06T08:29:05.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.459+0000 7fbf2f7fe640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbf24065430 0x7fbf240678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:05.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.459+0000 7fbf2f7fe640 1 -- 172.21.15.86:0/377220396 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbf280a2190 con 0x7fbf381011b0 2024-06-06T08:29:05.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.459+0000 7fbf2ffff640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbf24065430 0x7fbf240678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:05.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.460+0000 7fbf2ffff640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbf24065430 0x7fbf240678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbf38066210 tx=0x7fbf20008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:05.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.463+0000 7fbf2f7fe640 1 -- 172.21.15.86:0/377220396 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbf28070760 con 0x7fbf381011b0 2024-06-06T08:29:05.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.662+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbf3804b4e0 con 0x7fbf381011b0 2024-06-06T08:29:05.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.663+0000 7fbf2f7fe640 1 -- 172.21.15.86:0/377220396 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbf28073d80 con 0x7fbf381011b0 2024-06-06T08:29:05.662 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:05.662 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbf24065430 msgr2=0x7fbf240678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbf24065430 0x7fbf240678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbf38066210 tx=0x7fbf20008040 comp rx=0 tx=0).stop 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 msgr2=0x7fbf3810e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 0x7fbf3810e010 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbf28003080 tx=0x7fbf28009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 shutdown_connections 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf38101b50 0x7fbf3810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbf24065430 0x7fbf240678e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 --2- 172.21.15.86:0/377220396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf381011b0 0x7fbf3810e010 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 >> 172.21.15.86:0/377220396 conn(0x7fbf380f6da0 msgr2=0x7fbf380f7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 shutdown_connections 2024-06-06T08:29:05.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:05.665+0000 7fbf3d057640 1 -- 172.21.15.86:0/377220396 wait complete. 2024-06-06T08:29:06.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:06 smithi086 ceph-mon[29114]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:06.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:06 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/377220396' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:06.469 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:06 smithi154 ceph-mon[35428]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:06.469 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:06 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/377220396' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:08.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:08 smithi086 ceph-mon[29114]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:08.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:08 smithi154 ceph-mon[35428]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:09.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:09.648 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:10.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:10 smithi086 ceph-mon[29114]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:10.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:10 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:29:10.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:10 smithi154 ceph-mon[35428]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:10.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:10 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:29:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.046+0000 7fa018e5a640 1 -- 172.21.15.86:0/2104918180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 msgr2=0x7fa0141015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.046+0000 7fa018e5a640 1 --2- 172.21.15.86:0/2104918180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 0x7fa0141015a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa0040048d0 tx=0x7fa004009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.046+0000 7fa018e5a640 1 -- 172.21.15.86:0/2104918180 shutdown_connections 2024-06-06T08:29:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.046+0000 7fa018e5a640 1 --2- 172.21.15.86:0/2104918180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa014101b70 0x7fa0140fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.046+0000 7fa018e5a640 1 --2- 172.21.15.86:0/2104918180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 0x7fa0141015a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.046+0000 7fa018e5a640 1 -- 172.21.15.86:0/2104918180 >> 172.21.15.86:0/2104918180 conn(0x7fa0140f6dc0 msgr2=0x7fa0140f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.046+0000 7fa018e5a640 1 -- 172.21.15.86:0/2104918180 shutdown_connections 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.046+0000 7fa018e5a640 1 -- 172.21.15.86:0/2104918180 wait complete. 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa018e5a640 1 Processor -- start 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa018e5a640 1 -- start start 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa018e5a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 0x7fa01410e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa018e5a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa014101b70 0x7fa01410e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa018e5a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01410ec70 con 0x7fa0141011d0 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa018e5a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01410ede0 con 0x7fa014101b70 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa01259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 0x7fa01410e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa01259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 0x7fa01410e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37222/0 (socket says 172.21.15.86:37222) 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa01259c640 1 -- 172.21.15.86:0/175819429 learned_addr learned my addr 172.21.15.86:0/175819429 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa00bfff640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa014101b70 0x7fa01410e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa01259c640 1 -- 172.21.15.86:0/175819429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa014101b70 msgr2=0x7fa01410e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa01259c640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa014101b70 0x7fa01410e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.047+0000 7fa01259c640 1 -- 172.21.15.86:0/175819429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa004005040 con 0x7fa0141011d0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.048+0000 7fa01259c640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 0x7fa01410e050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa004003080 tx=0x7fa004009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.048+0000 7fa00affd640 1 -- 172.21.15.86:0/175819429 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa004038070 con 0x7fa0141011d0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.048+0000 7fa00affd640 1 -- 172.21.15.86:0/175819429 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa004008ee0 con 0x7fa0141011d0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.048+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0141170c0 con 0x7fa0141011d0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.048+0000 7fa00affd640 1 -- 172.21.15.86:0/175819429 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa00403d870 con 0x7fa0141011d0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.048+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa014117530 con 0x7fa0141011d0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.049+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0140fc940 con 0x7fa0141011d0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.049+0000 7fa00affd640 1 -- 172.21.15.86:0/175819429 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa004030020 con 0x7fa0141011d0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.050+0000 7fa00affd640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa000065430 0x7fa0000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.050+0000 7fa00affd640 1 -- 172.21.15.86:0/175819429 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa0040a21a0 con 0x7fa0141011d0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.050+0000 7fa00bfff640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa000065430 0x7fa0000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:11.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.050+0000 7fa00bfff640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa000065430 0x7fa0000678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa0140661d0 tx=0x7f9ffc009120 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:11.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.054+0000 7fa00affd640 1 -- 172.21.15.86:0/175819429 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa0040706d0 con 0x7fa0141011d0 2024-06-06T08:29:11.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.251+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa014065f80 con 0x7fa0141011d0 2024-06-06T08:29:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.252+0000 7fa00affd640 1 -- 172.21.15.86:0/175819429 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa004073cf0 con 0x7fa0141011d0 2024-06-06T08:29:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:11.251 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.253+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa000065430 msgr2=0x7fa0000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.253+0000 7fa018e5a640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa000065430 0x7fa0000678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa0140661d0 tx=0x7f9ffc009120 comp rx=0 tx=0).stop 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.253+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 msgr2=0x7fa01410e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.253+0000 7fa018e5a640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 0x7fa01410e050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa004003080 tx=0x7fa004009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.254+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 shutdown_connections 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.254+0000 7fa018e5a640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa014101b70 0x7fa01410e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.254+0000 7fa018e5a640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa000065430 0x7fa0000678e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.254+0000 7fa018e5a640 1 --2- 172.21.15.86:0/175819429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0141011d0 0x7fa01410e050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.254+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 >> 172.21.15.86:0/175819429 conn(0x7fa0140f6dc0 msgr2=0x7fa0140f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.254+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 shutdown_connections 2024-06-06T08:29:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:11.254+0000 7fa018e5a640 1 -- 172.21.15.86:0/175819429 wait complete. 2024-06-06T08:29:12.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:12 smithi086 ceph-mon[29114]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:12.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:12 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/175819429' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:12.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:12 smithi154 ceph-mon[35428]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:12.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:12 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/175819429' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:14.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:14 smithi086 ceph-mon[29114]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:14.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:14 smithi154 ceph-mon[35428]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:14.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:15.109 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:16.465 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:16 smithi086 ceph-mon[29114]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:16.601 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:16 smithi154 ceph-mon[35428]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:16.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.653+0000 7fd089433640 1 -- 172.21.15.86:0/2687338222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 msgr2=0x7fd084102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:16.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.653+0000 7fd089433640 1 --2- 172.21.15.86:0/2687338222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 0x7fd084102670 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009c50 tx=0x7fd07c004d10 comp rx=0 tx=0).stop 2024-06-06T08:29:16.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.653+0000 7fd089433640 1 -- 172.21.15.86:0/2687338222 shutdown_connections 2024-06-06T08:29:16.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.653+0000 7fd089433640 1 --2- 172.21.15.86:0/2687338222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd084102c40 0x7fd0840fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:16.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.653+0000 7fd089433640 1 --2- 172.21.15.86:0/2687338222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 0x7fd084102670 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:16.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.653+0000 7fd089433640 1 -- 172.21.15.86:0/2687338222 >> 172.21.15.86:0/2687338222 conn(0x7fd0840f7e70 msgr2=0x7fd0840fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:16.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.653+0000 7fd089433640 1 -- 172.21.15.86:0/2687338222 shutdown_connections 2024-06-06T08:29:16.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.653+0000 7fd089433640 1 -- 172.21.15.86:0/2687338222 wait complete. 2024-06-06T08:29:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd089433640 1 Processor -- start 2024-06-06T08:29:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd089433640 1 -- start start 2024-06-06T08:29:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd089433640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 0x7fd084115200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd089433640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd084102c40 0x7fd084115740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd089433640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd084115d40 con 0x7fd0841022a0 2024-06-06T08:29:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd089433640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08410f180 con 0x7fd084102c40 2024-06-06T08:29:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd082ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 0x7fd084115200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd082ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 0x7fd084115200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37252/0 (socket says 172.21.15.86:37252) 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd082ffd640 1 -- 172.21.15.86:0/2274005748 learned_addr learned my addr 172.21.15.86:0/2274005748 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd082ffd640 1 -- 172.21.15.86:0/2274005748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd084102c40 msgr2=0x7fd084115740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd0827fc640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd084102c40 0x7fd084115740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd082ffd640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd084102c40 0x7fd084115740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd082ffd640 1 -- 172.21.15.86:0/2274005748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0098b0 con 0x7fd0841022a0 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.654+0000 7fd0827fc640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd084102c40 0x7fd084115740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.655+0000 7fd082ffd640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 0x7fd084115200 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd07c004f80 tx=0x7fd07c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.655+0000 7fd07b7fe640 1 -- 172.21.15.86:0/2274005748 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd07c007d50 con 0x7fd0841022a0 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.655+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08410f400 con 0x7fd0841022a0 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.655+0000 7fd07b7fe640 1 -- 172.21.15.86:0/2274005748 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd07c032c00 con 0x7fd0841022a0 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.655+0000 7fd07b7fe640 1 -- 172.21.15.86:0/2274005748 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd07c03db90 con 0x7fd0841022a0 2024-06-06T08:29:16.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.655+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08410f8c0 con 0x7fd0841022a0 2024-06-06T08:29:16.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.656+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0840fda10 con 0x7fd0841022a0 2024-06-06T08:29:16.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.656+0000 7fd07b7fe640 1 -- 172.21.15.86:0/2274005748 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd07c02f030 con 0x7fd0841022a0 2024-06-06T08:29:16.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.657+0000 7fd07b7fe640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd070065430 0x7fd0700678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:16.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.657+0000 7fd07b7fe640 1 -- 172.21.15.86:0/2274005748 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd07c0a3700 con 0x7fd0841022a0 2024-06-06T08:29:16.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.657+0000 7fd0827fc640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd070065430 0x7fd0700678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:16.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.657+0000 7fd0827fc640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd070065430 0x7fd0700678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd084110ac0 tx=0x7fd06c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:16.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.661+0000 7fd07b7fe640 1 -- 172.21.15.86:0/2274005748 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd07c071bb0 con 0x7fd0841022a0 2024-06-06T08:29:16.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.867+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd08404b480 con 0x7fd0841022a0 2024-06-06T08:29:16.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.867+0000 7fd07b7fe640 1 -- 172.21.15.86:0/2274005748 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd07c04a100 con 0x7fd0841022a0 2024-06-06T08:29:16.866 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:16.866 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:16.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd070065430 msgr2=0x7fd0700678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:16.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd070065430 0x7fd0700678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd084110ac0 tx=0x7fd06c00b040 comp rx=0 tx=0).stop 2024-06-06T08:29:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 msgr2=0x7fd084115200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 0x7fd084115200 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd07c004f80 tx=0x7fd07c004d10 comp rx=0 tx=0).stop 2024-06-06T08:29:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 shutdown_connections 2024-06-06T08:29:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd084102c40 0x7fd084115740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd070065430 0x7fd0700678e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 --2- 172.21.15.86:0/2274005748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0841022a0 0x7fd084115200 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 >> 172.21.15.86:0/2274005748 conn(0x7fd0840f7e70 msgr2=0x7fd084071110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 shutdown_connections 2024-06-06T08:29:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:16.869+0000 7fd089433640 1 -- 172.21.15.86:0/2274005748 wait complete. 2024-06-06T08:29:17.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:17 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2274005748' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:17.666 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:17 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2274005748' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:18.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:18 smithi086 ceph-mon[29114]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:18.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:18 smithi154 ceph-mon[35428]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:20.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:20.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:20 smithi086 ceph-mon[29114]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:20.641 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:20.666 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:20 smithi154 ceph-mon[35428]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:22.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.080+0000 7f669b847640 1 -- 172.21.15.86:0/1342612359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 msgr2=0x7f66940faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:22.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.080+0000 7f669b847640 1 --2- 172.21.15.86:0/1342612359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 0x7f66940faec0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f66900048d0 tx=0x7f6690009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:22.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.080+0000 7f669b847640 1 -- 172.21.15.86:0/1342612359 shutdown_connections 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.080+0000 7f669b847640 1 --2- 172.21.15.86:0/1342612359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 0x7f66940faec0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.080+0000 7f669b847640 1 --2- 172.21.15.86:0/1342612359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66941023e0 0x7f66941027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.080+0000 7f669b847640 1 -- 172.21.15.86:0/1342612359 >> 172.21.15.86:0/1342612359 conn(0x7f6694070ab0 msgr2=0x7f6694070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.080+0000 7f669b847640 1 -- 172.21.15.86:0/1342612359 shutdown_connections 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.080+0000 7f669b847640 1 -- 172.21.15.86:0/1342612359 wait complete. 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f669b847640 1 Processor -- start 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f669b847640 1 -- start start 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f669b847640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 0x7f6694113180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f669b847640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66941023e0 0x7f66941136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f669b847640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6694113d50 con 0x7f66940faa50 2024-06-06T08:29:22.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f669b847640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669410d100 con 0x7f66941023e0 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f66995e3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 0x7f6694113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f66995e3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 0x7f6694113180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48152/0 (socket says 172.21.15.86:48152) 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f66995e3640 1 -- 172.21.15.86:0/1752328117 learned_addr learned my addr 172.21.15.86:0/1752328117 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.081+0000 7f6698de2640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66941023e0 0x7f66941136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.082+0000 7f66995e3640 1 -- 172.21.15.86:0/1752328117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66941023e0 msgr2=0x7f66941136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.082+0000 7f66995e3640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66941023e0 0x7f66941136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.082+0000 7f66995e3640 1 -- 172.21.15.86:0/1752328117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6690005040 con 0x7f66940faa50 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.082+0000 7f66995e3640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 0x7f6694113180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f668400b790 tx=0x7f66840091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.082+0000 7f6689ffb640 1 -- 172.21.15.86:0/1752328117 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f668400c700 con 0x7f66940faa50 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.082+0000 7f6689ffb640 1 -- 172.21.15.86:0/1752328117 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f668400cd30 con 0x7f66940faa50 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.082+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669410d3e0 con 0x7f66940faa50 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.082+0000 7f6689ffb640 1 -- 172.21.15.86:0/1752328117 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6684013370 con 0x7f66940faa50 2024-06-06T08:29:22.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.082+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669410d900 con 0x7f66940faa50 2024-06-06T08:29:22.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.083+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6694065550 con 0x7f66940faa50 2024-06-06T08:29:22.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.084+0000 7f6689ffb640 1 -- 172.21.15.86:0/1752328117 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f6684012070 con 0x7f66940faa50 2024-06-06T08:29:22.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.084+0000 7f6689ffb640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6680065430 0x7f66800678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:22.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.084+0000 7f6689ffb640 1 -- 172.21.15.86:0/1752328117 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6684081e60 con 0x7f66940faa50 2024-06-06T08:29:22.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.084+0000 7f6698de2640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6680065430 0x7f66800678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:22.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.084+0000 7f6698de2640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6680065430 0x7f66800678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6690032430 tx=0x7f6690009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:22.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.088+0000 7f6689ffb640 1 -- 172.21.15.86:0/1752328117 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6684050310 con 0x7f66940faa50 2024-06-06T08:29:22.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.287+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f66941095b0 con 0x7f66940faa50 2024-06-06T08:29:22.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:22 smithi086 ceph-mon[29114]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.287+0000 7f6689ffb640 1 -- 172.21.15.86:0/1752328117 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6684053930 con 0x7f66940faa50 2024-06-06T08:29:22.288 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:22.288 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6680065430 msgr2=0x7f66800678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6680065430 0x7f66800678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6690032430 tx=0x7f6690009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 msgr2=0x7f6694113180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 0x7f6694113180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f668400b790 tx=0x7f66840091d0 comp rx=0 tx=0).stop 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 shutdown_connections 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66941023e0 0x7f66941136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f6680065430 0x7f66800678e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 --2- 172.21.15.86:0/1752328117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940faa50 0x7f6694113180 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 >> 172.21.15.86:0/1752328117 conn(0x7f6694070ab0 msgr2=0x7f66940f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 shutdown_connections 2024-06-06T08:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:22.289+0000 7f669b847640 1 -- 172.21.15.86:0/1752328117 wait complete. 2024-06-06T08:29:22.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:22 smithi154 ceph-mon[35428]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:23.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:23 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1752328117' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:23.666 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:23 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1752328117' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:24.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:24 smithi086 ceph-mon[29114]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:24.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:24 smithi154 ceph-mon[35428]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:25.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:25 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:29:25.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:25 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:29:25.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:26.078 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:26.450 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:26 smithi086 ceph-mon[29114]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:26.601 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:26 smithi154 ceph-mon[35428]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:27.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.450+0000 7f12ba25a640 1 -- 172.21.15.86:0/1680637705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fa760 msgr2=0x7f12b40fab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.450+0000 7f12ba25a640 1 --2- 172.21.15.86:0/1680637705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fa760 0x7f12b40fab30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f12a4004900 tx=0x7f12a4009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 -- 172.21.15.86:0/1680637705 shutdown_connections 2024-06-06T08:29:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 --2- 172.21.15.86:0/1680637705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12b40fb100 0x7f12b4106140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 --2- 172.21.15.86:0/1680637705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fa760 0x7f12b40fab30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 -- 172.21.15.86:0/1680637705 >> 172.21.15.86:0/1680637705 conn(0x7f12b40724a0 msgr2=0x7f12b40728a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 -- 172.21.15.86:0/1680637705 shutdown_connections 2024-06-06T08:29:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 -- 172.21.15.86:0/1680637705 wait complete. 2024-06-06T08:29:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 Processor -- start 2024-06-06T08:29:27.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 -- start start 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12b40fa760 0x7f12b4102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fb100 0x7f12b41027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b4106410 con 0x7f12b40fb100 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.451+0000 7f12ba25a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b4102ce0 con 0x7f12b40fa760 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12b2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fb100 0x7f12b41027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12b2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fb100 0x7f12b41027a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60834/0 (socket says 172.21.15.86:60834) 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12b2ffd640 1 -- 172.21.15.86:0/2246298555 learned_addr learned my addr 172.21.15.86:0/2246298555 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12b37fe640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12b40fa760 0x7f12b4102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12b2ffd640 1 -- 172.21.15.86:0/2246298555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12b40fa760 msgr2=0x7f12b4102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12b2ffd640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12b40fa760 0x7f12b4102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12b2ffd640 1 -- 172.21.15.86:0/2246298555 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a4005040 con 0x7f12b40fb100 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12b2ffd640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fb100 0x7f12b41027a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f12a800b580 tx=0x7f12a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f129bfff640 1 -- 172.21.15.86:0/2246298555 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a800f6f0 con 0x7f12b40fb100 2024-06-06T08:29:27.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f129bfff640 1 -- 172.21.15.86:0/2246298555 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12a800fd20 con 0x7f12b40fb100 2024-06-06T08:29:27.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b4102fc0 con 0x7f12b40fb100 2024-06-06T08:29:27.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f129bfff640 1 -- 172.21.15.86:0/2246298555 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a80103a0 con 0x7f12b40fb100 2024-06-06T08:29:27.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.452+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b411ffb0 con 0x7f12b40fb100 2024-06-06T08:29:27.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.453+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b40fb810 con 0x7f12b40fb100 2024-06-06T08:29:27.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.454+0000 7f129bfff640 1 -- 172.21.15.86:0/2246298555 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f12a8011400 con 0x7f12b40fb100 2024-06-06T08:29:27.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.454+0000 7f129bfff640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f129c065480 0x7f129c067930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:27.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.454+0000 7f129bfff640 1 -- 172.21.15.86:0/2246298555 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f12a8082400 con 0x7f12b40fb100 2024-06-06T08:29:27.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.454+0000 7f12b37fe640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f129c065480 0x7f129c067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:27.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.455+0000 7f12b37fe640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f129c065480 0x7f129c067930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f12a4000c00 tx=0x7f12a4009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:27.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.458+0000 7f129bfff640 1 -- 172.21.15.86:0/2246298555 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f12a8050930 con 0x7f12b40fb100 2024-06-06T08:29:27.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.659+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f12b4072fb0 con 0x7f12b40fb100 2024-06-06T08:29:27.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.659+0000 7f129bfff640 1 -- 172.21.15.86:0/2246298555 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f12a8082080 con 0x7f12b40fb100 2024-06-06T08:29:27.659 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:27.659 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:27.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f129c065480 msgr2=0x7f129c067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:27.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f129c065480 0x7f129c067930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f12a4000c00 tx=0x7f12a4009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:27.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fb100 msgr2=0x7f12b41027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:27.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fb100 0x7f12b41027a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f12a800b580 tx=0x7f12a80091a0 comp rx=0 tx=0).stop 2024-06-06T08:29:27.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 shutdown_connections 2024-06-06T08:29:27.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f129c065480 0x7f129c067930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:27.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b40fb100 0x7f12b41027a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:27.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 --2- 172.21.15.86:0/2246298555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12b40fa760 0x7f12b4102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:27.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 >> 172.21.15.86:0/2246298555 conn(0x7f12b40724a0 msgr2=0x7f12b40f90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:27.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 shutdown_connections 2024-06-06T08:29:27.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:27.661+0000 7f12ba25a640 1 -- 172.21.15.86:0/2246298555 wait complete. 2024-06-06T08:29:28.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:28 smithi086 ceph-mon[29114]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:28.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:28 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2246298555' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:28.666 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:28 smithi154 ceph-mon[35428]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:28.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:28 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2246298555' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:30.585 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:30 smithi086 ceph-mon[29114]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:30.666 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:30 smithi154 ceph-mon[35428]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:31.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:31.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:31 smithi086 ceph-mon[29114]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:31.627 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:31.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:31 smithi154 ceph-mon[35428]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.255+0000 7f22953a1640 1 -- 172.21.15.86:0/2032553002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 msgr2=0x7f2290101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.255+0000 7f22953a1640 1 --2- 172.21.15.86:0/2032553002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 0x7f2290101580 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f22800048d0 tx=0x7f2280009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.255+0000 7f22953a1640 1 -- 172.21.15.86:0/2032553002 shutdown_connections 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.255+0000 7f22953a1640 1 --2- 172.21.15.86:0/2032553002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2290101b50 0x7f22900fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.255+0000 7f22953a1640 1 --2- 172.21.15.86:0/2032553002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 0x7f2290101580 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.255+0000 7f22953a1640 1 -- 172.21.15.86:0/2032553002 >> 172.21.15.86:0/2032553002 conn(0x7f22900f6d60 msgr2=0x7f22900f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.255+0000 7f22953a1640 1 -- 172.21.15.86:0/2032553002 shutdown_connections 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.255+0000 7f22953a1640 1 -- 172.21.15.86:0/2032553002 wait complete. 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f22953a1640 1 Processor -- start 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f22953a1640 1 -- start start 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f22953a1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 0x7f2290102030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f22953a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2290101b50 0x7f2290102570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f22953a1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2290102ab0 con 0x7f22901011b0 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f22953a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2290102c20 con 0x7f2290101b50 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f228effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 0x7f2290102030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f228effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 0x7f2290102030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60844/0 (socket says 172.21.15.86:60844) 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f228effd640 1 -- 172.21.15.86:0/856070291 learned_addr learned my addr 172.21.15.86:0/856070291 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f228e7fc640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2290101b50 0x7f2290102570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f228effd640 1 -- 172.21.15.86:0/856070291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2290101b50 msgr2=0x7f2290102570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f228effd640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2290101b50 0x7f2290102570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f228effd640 1 -- 172.21.15.86:0/856070291 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2280005040 con 0x7f22901011b0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.256+0000 7f228effd640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 0x7f2290102030 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2280003080 tx=0x7f2280009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.257+0000 7f22777fe640 1 -- 172.21.15.86:0/856070291 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2280038070 con 0x7f22901011b0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.257+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2290102e50 con 0x7f22901011b0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.257+0000 7f22777fe640 1 -- 172.21.15.86:0/856070291 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2280008ee0 con 0x7f22901011b0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.257+0000 7f22777fe640 1 -- 172.21.15.86:0/856070291 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f228003d870 con 0x7f22901011b0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.257+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22901196b0 con 0x7f22901011b0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.258+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22900fc920 con 0x7f22901011b0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.258+0000 7f22777fe640 1 -- 172.21.15.86:0/856070291 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f2280030020 con 0x7f22901011b0 2024-06-06T08:29:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.258+0000 7f22777fe640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2278065430 0x7f22780678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.258+0000 7f22777fe640 1 -- 172.21.15.86:0/856070291 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f22800a2250 con 0x7f22901011b0 2024-06-06T08:29:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.259+0000 7f228e7fc640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2278065430 0x7f22780678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.259+0000 7f228e7fc640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2278065430 0x7f22780678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2284007fd0 tx=0x7f2284009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.263+0000 7f22777fe640 1 -- 172.21.15.86:0/856070291 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2280070780 con 0x7f22901011b0 2024-06-06T08:29:33.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.461+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f22900fa9a0 con 0x7f22901011b0 2024-06-06T08:29:33.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.461+0000 7f22777fe640 1 -- 172.21.15.86:0/856070291 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2280073da0 con 0x7f22901011b0 2024-06-06T08:29:33.461 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:33.461 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:33.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2278065430 msgr2=0x7f22780678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2278065430 0x7f22780678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2284007fd0 tx=0x7f2284009040 comp rx=0 tx=0).stop 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 msgr2=0x7f2290102030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 0x7f2290102030 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2280003080 tx=0x7f2280009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 shutdown_connections 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2290101b50 0x7f2290102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f2278065430 0x7f22780678e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 --2- 172.21.15.86:0/856070291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22901011b0 0x7f2290102030 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 >> 172.21.15.86:0/856070291 conn(0x7f22900f6d60 msgr2=0x7f22900f83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 shutdown_connections 2024-06-06T08:29:33.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:33.463+0000 7f22953a1640 1 -- 172.21.15.86:0/856070291 wait complete. 2024-06-06T08:29:34.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:33 smithi086 ceph-mon[29114]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:34.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:33 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/856070291' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:34.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:33 smithi154 ceph-mon[35428]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:34.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:33 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/856070291' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:36.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:35 smithi086 ceph-mon[29114]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:36.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:35 smithi154 ceph-mon[35428]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:37.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:37.413 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:37.800 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:37 smithi086 ceph-mon[29114]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:38.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:37 smithi154 ceph-mon[35428]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.959+0000 7fe08d66f640 1 -- 172.21.15.86:0/1161089406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0880fdea0 msgr2=0x7fe0880fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.959+0000 7fe08d66f640 1 --2- 172.21.15.86:0/1161089406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0880fdea0 0x7fe0880fe310 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe0700048d0 tx=0x7fe070009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.960+0000 7fe08d66f640 1 -- 172.21.15.86:0/1161089406 shutdown_connections 2024-06-06T08:29:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.960+0000 7fe08d66f640 1 --2- 172.21.15.86:0/1161089406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0880fdea0 0x7fe0880fe310 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.960+0000 7fe08d66f640 1 --2- 172.21.15.86:0/1161089406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe088065030 0x7fe088065400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:38.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.960+0000 7fe08d66f640 1 -- 172.21.15.86:0/1161089406 >> 172.21.15.86:0/1161089406 conn(0x7fe088072110 msgr2=0x7fe088072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:38.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.960+0000 7fe08d66f640 1 -- 172.21.15.86:0/1161089406 shutdown_connections 2024-06-06T08:29:38.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.960+0000 7fe08d66f640 1 -- 172.21.15.86:0/1161089406 wait complete. 2024-06-06T08:29:38.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.960+0000 7fe08d66f640 1 Processor -- start 2024-06-06T08:29:38.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.960+0000 7fe08d66f640 1 -- start start 2024-06-06T08:29:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe08d66f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe088065030 0x7fe088113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe08d66f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0880fdea0 0x7fe0881135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe08d66f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe088113bb0 con 0x7fe088065030 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe08d66f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08810d040 con 0x7fe0880fdea0 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe086ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe088065030 0x7fe088113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe086ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe088065030 0x7fe088113070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37508/0 (socket says 172.21.15.86:37508) 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe086ffd640 1 -- 172.21.15.86:0/2971666216 learned_addr learned my addr 172.21.15.86:0/2971666216 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe07e7fc640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0880fdea0 0x7fe0881135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe086ffd640 1 -- 172.21.15.86:0/2971666216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0880fdea0 msgr2=0x7fe0881135b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe086ffd640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0880fdea0 0x7fe0881135b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe086ffd640 1 -- 172.21.15.86:0/2971666216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070005040 con 0x7fe088065030 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe07e7fc640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0880fdea0 0x7fe0881135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe086ffd640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe088065030 0x7fe088113070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe078002a50 tx=0x7fe078004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe07ffff640 1 -- 172.21.15.86:0/2971666216 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe078007c30 con 0x7fe088065030 2024-06-06T08:29:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08810d320 con 0x7fe088065030 2024-06-06T08:29:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe07ffff640 1 -- 172.21.15.86:0/2971666216 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe07800f040 con 0x7fe088065030 2024-06-06T08:29:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe07ffff640 1 -- 172.21.15.86:0/2971666216 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe07800db50 con 0x7fe088065030 2024-06-06T08:29:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.961+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08810d840 con 0x7fe088065030 2024-06-06T08:29:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.962+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0880fda40 con 0x7fe088065030 2024-06-06T08:29:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.963+0000 7fe07ffff640 1 -- 172.21.15.86:0/2971666216 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe07801c400 con 0x7fe088065030 2024-06-06T08:29:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.963+0000 7fe07ffff640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe074065430 0x7fe0740678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.963+0000 7fe07ffff640 1 -- 172.21.15.86:0/2971666216 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe078081da0 con 0x7fe088065030 2024-06-06T08:29:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.963+0000 7fe07e7fc640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe074065430 0x7fe0740678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.964+0000 7fe07e7fc640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe074065430 0x7fe0740678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe08810e930 tx=0x7fe070035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:38.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:38.968+0000 7fe07ffff640 1 -- 172.21.15.86:0/2971666216 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe07804cdf0 con 0x7fe088065030 2024-06-06T08:29:39.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.187+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe08804b480 con 0x7fe088065030 2024-06-06T08:29:39.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.187+0000 7fe07ffff640 1 -- 172.21.15.86:0/2971666216 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe078050250 con 0x7fe088065030 2024-06-06T08:29:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:39.187 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:39.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe074065430 msgr2=0x7fe0740678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:39.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe074065430 0x7fe0740678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe08810e930 tx=0x7fe070035040 comp rx=0 tx=0).stop 2024-06-06T08:29:39.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe088065030 msgr2=0x7fe088113070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:39.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe088065030 0x7fe088113070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe078002a50 tx=0x7fe078004600 comp rx=0 tx=0).stop 2024-06-06T08:29:39.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 shutdown_connections 2024-06-06T08:29:39.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0880fdea0 0x7fe0881135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:39.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe074065430 0x7fe0740678e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:39.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 --2- 172.21.15.86:0/2971666216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe088065030 0x7fe088113070 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:39.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 >> 172.21.15.86:0/2971666216 conn(0x7fe088072110 msgr2=0x7fe088105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:39.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 shutdown_connections 2024-06-06T08:29:39.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:39.189+0000 7fe08d66f640 1 -- 172.21.15.86:0/2971666216 wait complete. 2024-06-06T08:29:39.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:39 smithi086 ceph-mon[29114]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:39.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:39 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2971666216' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:39.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:39 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:29:40.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:39 smithi154 ceph-mon[35428]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:40.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:39 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2971666216' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:40.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:39 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:29:42.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:41 smithi086 ceph-mon[29114]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:42.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:41 smithi154 ceph-mon[35428]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:42.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:43.053 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:44.000 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:43 smithi086 ceph-mon[29114]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:44.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:43 smithi154 ceph-mon[35428]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:44.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.466+0000 7eff39267640 1 -- 172.21.15.86:0/1848774994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 msgr2=0x7eff340fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:44.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.466+0000 7eff39267640 1 --2- 172.21.15.86:0/1848774994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 0x7eff340fb200 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0048d0 tx=0x7eff1c009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:44.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.466+0000 7eff39267640 1 -- 172.21.15.86:0/1848774994 shutdown_connections 2024-06-06T08:29:44.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.466+0000 7eff39267640 1 --2- 172.21.15.86:0/1848774994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 0x7eff340fb200 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:44.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.466+0000 7eff39267640 1 --2- 172.21.15.86:0/1848774994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341011d0 0x7eff341015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:44.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.466+0000 7eff39267640 1 -- 172.21.15.86:0/1848774994 >> 172.21.15.86:0/1848774994 conn(0x7eff340f6dc0 msgr2=0x7eff340f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:44.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.466+0000 7eff39267640 1 -- 172.21.15.86:0/1848774994 shutdown_connections 2024-06-06T08:29:44.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.466+0000 7eff39267640 1 -- 172.21.15.86:0/1848774994 wait complete. 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff39267640 1 Processor -- start 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff39267640 1 -- start start 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff39267640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341011d0 0x7eff34111f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff39267640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 0x7eff34112460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff39267640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff34112a60 con 0x7eff34101b70 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff39267640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3410bea0 con 0x7eff341011d0 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff2a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 0x7eff34112460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff2a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 0x7eff34112460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37532/0 (socket says 172.21.15.86:37532) 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff2a7fc640 1 -- 172.21.15.86:0/1065753145 learned_addr learned my addr 172.21.15.86:0/1065753145 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff32ffd640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341011d0 0x7eff34111f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.467+0000 7eff2a7fc640 1 -- 172.21.15.86:0/1065753145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341011d0 msgr2=0x7eff34111f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.468+0000 7eff2a7fc640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341011d0 0x7eff34111f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.468+0000 7eff2a7fc640 1 -- 172.21.15.86:0/1065753145 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c005040 con 0x7eff34101b70 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.468+0000 7eff2a7fc640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 0x7eff34112460 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0043d0 tx=0x7eff1c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:44.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.468+0000 7eff2bfff640 1 -- 172.21.15.86:0/1065753145 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff1c038070 con 0x7eff34101b70 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.468+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3410c120 con 0x7eff34101b70 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.468+0000 7eff2bfff640 1 -- 172.21.15.86:0/1065753145 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff1c032970 con 0x7eff34101b70 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.468+0000 7eff2bfff640 1 -- 172.21.15.86:0/1065753145 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff1c03da60 con 0x7eff34101b70 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.468+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3410c660 con 0x7eff34101b70 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.469+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff340fc940 con 0x7eff34101b70 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.469+0000 7eff2bfff640 1 -- 172.21.15.86:0/1065753145 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7eff1c030020 con 0x7eff34101b70 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.470+0000 7eff2bfff640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7eff20065430 0x7eff200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.470+0000 7eff32ffd640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7eff20065430 0x7eff200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.470+0000 7eff2bfff640 1 -- 172.21.15.86:0/1065753145 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7eff1c0a25d0 con 0x7eff34101b70 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.470+0000 7eff32ffd640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7eff20065430 0x7eff200678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7eff24004ed0 tx=0x7eff2400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:44.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.474+0000 7eff2bfff640 1 -- 172.21.15.86:0/1065753145 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7eff1c070a80 con 0x7eff34101b70 2024-06-06T08:29:44.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.672+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7eff340fa9c0 con 0x7eff34101b70 2024-06-06T08:29:44.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.672+0000 7eff2bfff640 1 -- 172.21.15.86:0/1065753145 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7eff1c0740a0 con 0x7eff34101b70 2024-06-06T08:29:44.672 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:44.672 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.674+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7eff20065430 msgr2=0x7eff200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.674+0000 7eff39267640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7eff20065430 0x7eff200678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7eff24004ed0 tx=0x7eff2400b040 comp rx=0 tx=0).stop 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.674+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 msgr2=0x7eff34112460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.674+0000 7eff39267640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 0x7eff34112460 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0043d0 tx=0x7eff1c009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.675+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 shutdown_connections 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.675+0000 7eff39267640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7eff20065430 0x7eff200678e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.675+0000 7eff39267640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff34101b70 0x7eff34112460 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.675+0000 7eff39267640 1 --2- 172.21.15.86:0/1065753145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341011d0 0x7eff34111f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.675+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 >> 172.21.15.86:0/1065753145 conn(0x7eff340f6dc0 msgr2=0x7eff340f7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.675+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 shutdown_connections 2024-06-06T08:29:44.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:44.675+0000 7eff39267640 1 -- 172.21.15.86:0/1065753145 wait complete. 2024-06-06T08:29:45.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:44 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1065753145' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:45.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:44 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1065753145' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:46.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:45 smithi086 ceph-mon[29114]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:46.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:45 smithi154 ceph-mon[35428]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:48.085 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:47 smithi086 ceph-mon[29114]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:48.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:47 smithi154 ceph-mon[35428]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:48.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:48.952 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:29:48.952 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:29:48.952 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:29:48.952 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:29:49.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:29:49.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:29:49.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:29:49.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:29:49.867 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:49 smithi086 ceph-mon[29114]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:50.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.060+0000 7f38f24a3640 1 -- 172.21.15.86:0/1960193603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec102c70 msgr2=0x7f38ec0fc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:50.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.060+0000 7f38f24a3640 1 --2- 172.21.15.86:0/1960193603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec102c70 0x7f38ec0fc300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f38e00048d0 tx=0x7f38e0009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:50.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.060+0000 7f38f24a3640 1 -- 172.21.15.86:0/1960193603 shutdown_connections 2024-06-06T08:29:50.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.060+0000 7f38f24a3640 1 --2- 172.21.15.86:0/1960193603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec102c70 0x7f38ec0fc300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:50.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.060+0000 7f38f24a3640 1 --2- 172.21.15.86:0/1960193603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38ec1022d0 0x7f38ec1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.060+0000 7f38f24a3640 1 -- 172.21.15.86:0/1960193603 >> 172.21.15.86:0/1960193603 conn(0x7f38ec0f7e80 msgr2=0x7f38ec0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.060+0000 7f38f24a3640 1 -- 172.21.15.86:0/1960193603 shutdown_connections 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.060+0000 7f38f24a3640 1 -- 172.21.15.86:0/1960193603 wait complete. 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38f24a3640 1 Processor -- start 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38f24a3640 1 -- start start 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38f24a3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec1022d0 0x7f38ec113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38f24a3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38ec102c70 0x7f38ec113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38f24a3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec113b60 con 0x7f38ec1022d0 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38f24a3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec10cfa0 con 0x7f38ec102c70 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38ebfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec1022d0 0x7f38ec113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38ebfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec1022d0 0x7f38ec113020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:41364/0 (socket says 172.21.15.86:41364) 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38ebfff640 1 -- 172.21.15.86:0/1391138853 learned_addr learned my addr 172.21.15.86:0/1391138853 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:50.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.061+0000 7f38eb7fe640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38ec102c70 0x7f38ec113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.062+0000 7f38ebfff640 1 -- 172.21.15.86:0/1391138853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38ec102c70 msgr2=0x7f38ec113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.062+0000 7f38ebfff640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38ec102c70 0x7f38ec113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.062+0000 7f38ebfff640 1 -- 172.21.15.86:0/1391138853 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e0005040 con 0x7f38ec1022d0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.062+0000 7f38ebfff640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec1022d0 0x7f38ec113020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f38dc002a50 tx=0x7f38dc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.062+0000 7f38e8ff9640 1 -- 172.21.15.86:0/1391138853 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38dc007c30 con 0x7f38ec1022d0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.062+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ec10d280 con 0x7f38ec1022d0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.062+0000 7f38e8ff9640 1 -- 172.21.15.86:0/1391138853 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38dc00f040 con 0x7f38ec1022d0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.062+0000 7f38e8ff9640 1 -- 172.21.15.86:0/1391138853 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38dc00dad0 con 0x7f38ec1022d0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.062+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ec10d7f0 con 0x7f38ec1022d0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.063+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ec0fda40 con 0x7f38ec1022d0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.063+0000 7f38e8ff9640 1 -- 172.21.15.86:0/1391138853 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f38dc00dd40 con 0x7f38ec1022d0 2024-06-06T08:29:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.064+0000 7f38e8ff9640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f38d4065430 0x7f38d40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:50.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.064+0000 7f38eb7fe640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f38d4065430 0x7f38d40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:50.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.064+0000 7f38e8ff9640 1 -- 172.21.15.86:0/1391138853 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f38dc081c90 con 0x7f38ec1022d0 2024-06-06T08:29:50.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.064+0000 7f38eb7fe640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f38d4065430 0x7f38d40678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009f80 tx=0x7f38e0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.068+0000 7f38e8ff9640 1 -- 172.21.15.86:0/1391138853 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f38dc04cdf0 con 0x7f38ec1022d0 2024-06-06T08:29:50.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:49 smithi154 ceph-mon[35428]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:50.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.264+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f38ec109430 con 0x7f38ec1022d0 2024-06-06T08:29:50.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.265+0000 7f38e8ff9640 1 -- 172.21.15.86:0/1391138853 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f38dc0501f0 con 0x7f38ec1022d0 2024-06-06T08:29:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:50.264 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f38d4065430 msgr2=0x7f38d40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f38d4065430 0x7f38d40678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009f80 tx=0x7f38e0035040 comp rx=0 tx=0).stop 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec1022d0 msgr2=0x7f38ec113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec1022d0 0x7f38ec113020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f38dc002a50 tx=0x7f38dc004600 comp rx=0 tx=0).stop 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 shutdown_connections 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38ec102c70 0x7f38ec113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f38d4065430 0x7f38d40678e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 --2- 172.21.15.86:0/1391138853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ec1022d0 0x7f38ec113020 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 >> 172.21.15.86:0/1391138853 conn(0x7f38ec0f7e80 msgr2=0x7f38ec0f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 shutdown_connections 2024-06-06T08:29:50.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:50.267+0000 7f38f24a3640 1 -- 172.21.15.86:0/1391138853 wait complete. 2024-06-06T08:29:50.962 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:50 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1391138853' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:51.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:50 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1391138853' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:52.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:51 smithi086 ceph-mon[29114]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:52.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:51 smithi154 ceph-mon[35428]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:53.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:54.015 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:53 smithi086 ceph-mon[29114]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:54.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:53 smithi154 ceph-mon[35428]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:54.254 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:29:55.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:29:55.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:29:55.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.566+0000 7fe719369640 1 -- 172.21.15.86:0/184778721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 msgr2=0x7fe714101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.566+0000 7fe719369640 1 --2- 172.21.15.86:0/184778721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 0x7fe714101580 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe7040048d0 tx=0x7fe704009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.566+0000 7fe719369640 1 -- 172.21.15.86:0/184778721 shutdown_connections 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.566+0000 7fe719369640 1 --2- 172.21.15.86:0/184778721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe714101b50 0x7fe7140fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.566+0000 7fe719369640 1 --2- 172.21.15.86:0/184778721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 0x7fe714101580 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.566+0000 7fe719369640 1 -- 172.21.15.86:0/184778721 >> 172.21.15.86:0/184778721 conn(0x7fe7140f6d60 msgr2=0x7fe7140f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.566+0000 7fe719369640 1 -- 172.21.15.86:0/184778721 shutdown_connections 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.566+0000 7fe719369640 1 -- 172.21.15.86:0/184778721 wait complete. 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe719369640 1 Processor -- start 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe719369640 1 -- start start 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe719369640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 0x7fe7141130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe719369640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe714101b50 0x7fe714113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe719369640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe714113c90 con 0x7fe7141011b0 2024-06-06T08:29:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe719369640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71410d080 con 0x7fe714101b50 2024-06-06T08:29:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe712ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 0x7fe7141130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe712ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 0x7fe7141130c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:41382/0 (socket says 172.21.15.86:41382) 2024-06-06T08:29:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe712ffd640 1 -- 172.21.15.86:0/462875611 learned_addr learned my addr 172.21.15.86:0/462875611 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:29:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.567+0000 7fe70bfff640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe714101b50 0x7fe714113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.568+0000 7fe712ffd640 1 -- 172.21.15.86:0/462875611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe714101b50 msgr2=0x7fe714113600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.568+0000 7fe712ffd640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe714101b50 0x7fe714113600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.568+0000 7fe712ffd640 1 -- 172.21.15.86:0/462875611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe704005040 con 0x7fe7141011b0 2024-06-06T08:29:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.568+0000 7fe712ffd640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 0x7fe7141130c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe704003080 tx=0x7fe704009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:55.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.568+0000 7fe70b7fe640 1 -- 172.21.15.86:0/462875611 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe704038070 con 0x7fe7141011b0 2024-06-06T08:29:55.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.568+0000 7fe70b7fe640 1 -- 172.21.15.86:0/462875611 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe704008ee0 con 0x7fe7141011b0 2024-06-06T08:29:55.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.568+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71410d300 con 0x7fe7141011b0 2024-06-06T08:29:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.568+0000 7fe70b7fe640 1 -- 172.21.15.86:0/462875611 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe70403d870 con 0x7fe7141011b0 2024-06-06T08:29:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.568+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71410d7c0 con 0x7fe7141011b0 2024-06-06T08:29:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.569+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7140fc920 con 0x7fe7141011b0 2024-06-06T08:29:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.569+0000 7fe70b7fe640 1 -- 172.21.15.86:0/462875611 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe704030020 con 0x7fe7141011b0 2024-06-06T08:29:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.570+0000 7fe70b7fe640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe700065430 0x7fe7000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:29:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.570+0000 7fe70b7fe640 1 -- 172.21.15.86:0/462875611 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe7040a2020 con 0x7fe7141011b0 2024-06-06T08:29:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.570+0000 7fe70bfff640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe700065430 0x7fe7000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:29:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.570+0000 7fe70bfff640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe700065430 0x7fe7000678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe71410e9c0 tx=0x7fe6fc009120 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:29:55.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.574+0000 7fe70b7fe640 1 -- 172.21.15.86:0/462875611 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe7040705f0 con 0x7fe7141011b0 2024-06-06T08:29:55.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.771+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe7140fa9a0 con 0x7fe7141011b0 2024-06-06T08:29:55.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.771+0000 7fe70b7fe640 1 -- 172.21.15.86:0/462875611 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe704073c10 con 0x7fe7141011b0 2024-06-06T08:29:55.771 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:29:55.771 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe700065430 msgr2=0x7fe7000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe700065430 0x7fe7000678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe71410e9c0 tx=0x7fe6fc009120 comp rx=0 tx=0).stop 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 msgr2=0x7fe7141130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 0x7fe7141130c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe704003080 tx=0x7fe704009da0 comp rx=0 tx=0).stop 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 shutdown_connections 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe714101b50 0x7fe714113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe700065430 0x7fe7000678e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 --2- 172.21.15.86:0/462875611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7141011b0 0x7fe7141130c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 >> 172.21.15.86:0/462875611 conn(0x7fe7140f6d60 msgr2=0x7fe7140f7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 shutdown_connections 2024-06-06T08:29:55.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:29:55.773+0000 7fe719369640 1 -- 172.21.15.86:0/462875611 wait complete. 2024-06-06T08:29:56.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:55 smithi086 ceph-mon[29114]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:56.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:55 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/462875611' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:56.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:55 smithi154 ceph-mon[35428]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:56.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:55 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/462875611' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:29:58.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:57 smithi086 ceph-mon[29114]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:58.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:57 smithi154 ceph-mon[35428]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:29:59.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:29:59.600 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:00.029 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:29:59 smithi086 ceph-mon[29114]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:00.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:29:59 smithi154 ceph-mon[35428]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:00.834 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:00 smithi086 ceph-mon[29114]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T08:30:01.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.064+0000 7f45d6a1a640 1 -- 172.21.15.86:0/1141459123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d01011b0 msgr2=0x7f45d0101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:01.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.064+0000 7f45d6a1a640 1 --2- 172.21.15.86:0/1141459123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d01011b0 0x7f45d0101580 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f45c00048d0 tx=0x7f45c0009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:01.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.064+0000 7f45d6a1a640 1 -- 172.21.15.86:0/1141459123 shutdown_connections 2024-06-06T08:30:01.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.064+0000 7f45d6a1a640 1 --2- 172.21.15.86:0/1141459123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d0101b50 0x7f45d00fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:01.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.064+0000 7f45d6a1a640 1 --2- 172.21.15.86:0/1141459123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d01011b0 0x7f45d0101580 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:01.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.064+0000 7f45d6a1a640 1 -- 172.21.15.86:0/1141459123 >> 172.21.15.86:0/1141459123 conn(0x7f45d00f6d60 msgr2=0x7f45d00f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.064+0000 7f45d6a1a640 1 -- 172.21.15.86:0/1141459123 shutdown_connections 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.064+0000 7f45d6a1a640 1 -- 172.21.15.86:0/1141459123 wait complete. 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45d6a1a640 1 Processor -- start 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45d6a1a640 1 -- start start 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45d6a1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d01011b0 0x7f45d0111f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45d6a1a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d0101b50 0x7f45d0112460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45d6a1a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d0112a60 con 0x7f45d0101b50 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45d6a1a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d010bea0 con 0x7f45d01011b0 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45cf7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d0101b50 0x7f45d0112460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45cf7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d0101b50 0x7f45d0112460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:41352/0 (socket says 172.21.15.86:41352) 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45cf7fe640 1 -- 172.21.15.86:0/2351959570 learned_addr learned my addr 172.21.15.86:0/2351959570 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45cffff640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d01011b0 0x7f45d0111f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45cf7fe640 1 -- 172.21.15.86:0/2351959570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d01011b0 msgr2=0x7f45d0111f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45cf7fe640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d01011b0 0x7f45d0111f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.065+0000 7f45cf7fe640 1 -- 172.21.15.86:0/2351959570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c0005040 con 0x7f45d0101b50 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.066+0000 7f45cf7fe640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d0101b50 0x7f45d0112460 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c830 tx=0x7f45c4004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.066+0000 7f45ccff9640 1 -- 172.21.15.86:0/2351959570 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45c400a7d0 con 0x7f45d0101b50 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.066+0000 7f45ccff9640 1 -- 172.21.15.86:0/2351959570 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45c400ae00 con 0x7f45d0101b50 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.066+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d010c180 con 0x7f45d0101b50 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.066+0000 7f45ccff9640 1 -- 172.21.15.86:0/2351959570 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45c40113a0 con 0x7f45d0101b50 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.066+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d010c720 con 0x7f45d0101b50 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.067+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d00fc920 con 0x7f45d0101b50 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.067+0000 7f45ccff9640 1 -- 172.21.15.86:0/2351959570 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f45c4012400 con 0x7f45d0101b50 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.068+0000 7f45ccff9640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f45b8065430 0x7f45b80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.068+0000 7f45ccff9640 1 -- 172.21.15.86:0/2351959570 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f45c40825d0 con 0x7f45d0101b50 2024-06-06T08:30:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.068+0000 7f45cffff640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f45b8065430 0x7f45b80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:01.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.068+0000 7f45cffff640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f45b8065430 0x7f45b80678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45c0032430 tx=0x7f45c0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:01.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.072+0000 7f45ccff9640 1 -- 172.21.15.86:0/2351959570 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f45c4050b00 con 0x7f45d0101b50 2024-06-06T08:30:01.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:00 smithi154 ceph-mon[35428]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T08:30:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.274+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f45d00fa9a0 con 0x7f45d0101b50 2024-06-06T08:30:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.274+0000 7f45ccff9640 1 -- 172.21.15.86:0/2351959570 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f45c4054120 con 0x7f45d0101b50 2024-06-06T08:30:01.274 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.276+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f45b8065430 msgr2=0x7f45b80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.276+0000 7f45d6a1a640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f45b8065430 0x7f45b80678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45c0032430 tx=0x7f45c0035040 comp rx=0 tx=0).stop 2024-06-06T08:30:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.276+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d0101b50 msgr2=0x7f45d0112460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.276+0000 7f45d6a1a640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d0101b50 0x7f45d0112460 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c830 tx=0x7f45c4004430 comp rx=0 tx=0).stop 2024-06-06T08:30:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.276+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 shutdown_connections 2024-06-06T08:30:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.276+0000 7f45d6a1a640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f45b8065430 0x7f45b80678e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.276+0000 7f45d6a1a640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d0101b50 0x7f45d0112460 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.276+0000 7f45d6a1a640 1 --2- 172.21.15.86:0/2351959570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d01011b0 0x7f45d0111f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.276+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 >> 172.21.15.86:0/2351959570 conn(0x7f45d00f6d60 msgr2=0x7f45d00f7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.277+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 shutdown_connections 2024-06-06T08:30:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:01.277+0000 7f45d6a1a640 1 -- 172.21.15.86:0/2351959570 wait complete. 2024-06-06T08:30:02.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:01 smithi086 ceph-mon[29114]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:02.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:01 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2351959570' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:02.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:01 smithi154 ceph-mon[35428]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:02.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:01 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2351959570' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:04.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:03 smithi086 ceph-mon[29114]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:04.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:03 smithi154 ceph-mon[35428]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:04.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:06.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:05 smithi086 ceph-mon[29114]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:06.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:05 smithi154 ceph-mon[35428]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:06.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.575+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/2691407624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 msgr2=0x7fdd740fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:06.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.575+0000 7fdd7a8ec640 1 --2- 172.21.15.86:0/2691407624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 0x7fdd740fb940 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdd640048d0 tx=0x7fdd64009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:06.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.575+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/2691407624 shutdown_connections 2024-06-06T08:30:06.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.575+0000 7fdd7a8ec640 1 --2- 172.21.15.86:0/2691407624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd740fbe80 0x7fdd740fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.575+0000 7fdd7a8ec640 1 --2- 172.21.15.86:0/2691407624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 0x7fdd740fb940 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.575+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/2691407624 >> 172.21.15.86:0/2691407624 conn(0x7fdd74072110 msgr2=0x7fdd74072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.575+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/2691407624 shutdown_connections 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.576+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/2691407624 wait complete. 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.576+0000 7fdd7a8ec640 1 Processor -- start 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.576+0000 7fdd7a8ec640 1 -- start start 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.576+0000 7fdd7a8ec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 0x7fdd7410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.576+0000 7fdd7a8ec640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd740fbe80 0x7fdd7410e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.576+0000 7fdd7a8ec640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7410eb70 con 0x7fdd740fb570 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.576+0000 7fdd7a8ec640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7410ece0 con 0x7fdd740fbe80 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.576+0000 7fdd73fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 0x7fdd7410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:06.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd73fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 0x7fdd7410dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:41378/0 (socket says 172.21.15.86:41378) 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd73fff640 1 -- 172.21.15.86:0/3487240457 learned_addr learned my addr 172.21.15.86:0/3487240457 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd737fe640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd740fbe80 0x7fdd7410e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd73fff640 1 -- 172.21.15.86:0/3487240457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd740fbe80 msgr2=0x7fdd7410e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd73fff640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd740fbe80 0x7fdd7410e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd73fff640 1 -- 172.21.15.86:0/3487240457 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64005040 con 0x7fdd740fb570 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd737fe640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd740fbe80 0x7fdd7410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd73fff640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 0x7fdd7410dfe0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdd64003080 tx=0x7fdd64009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd70ff9640 1 -- 172.21.15.86:0/3487240457 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd64038070 con 0x7fdd740fb570 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd70ff9640 1 -- 172.21.15.86:0/3487240457 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd64008ee0 con 0x7fdd740fb570 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd70ff9640 1 -- 172.21.15.86:0/3487240457 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd6403d870 con 0x7fdd740fb570 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd74117050 con 0x7fdd740fb570 2024-06-06T08:30:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.577+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd741174b0 con 0x7fdd740fb570 2024-06-06T08:30:06.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.578+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd74072630 con 0x7fdd740fb570 2024-06-06T08:30:06.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.579+0000 7fdd70ff9640 1 -- 172.21.15.86:0/3487240457 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fdd64030020 con 0x7fdd740fb570 2024-06-06T08:30:06.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.579+0000 7fdd70ff9640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd60065430 0x7fdd600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:06.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.579+0000 7fdd737fe640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd60065430 0x7fdd600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:06.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.579+0000 7fdd70ff9640 1 -- 172.21.15.86:0/3487240457 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdd640a2150 con 0x7fdd740fb570 2024-06-06T08:30:06.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.580+0000 7fdd737fe640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd60065430 0x7fdd600678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdd740656e0 tx=0x7fdd5c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:06.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.583+0000 7fdd70ff9640 1 -- 172.21.15.86:0/3487240457 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdd64070720 con 0x7fdd740fb570 2024-06-06T08:30:06.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.780+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdd740fc920 con 0x7fdd740fb570 2024-06-06T08:30:06.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.780+0000 7fdd70ff9640 1 -- 172.21.15.86:0/3487240457 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdd64073d40 con 0x7fdd740fb570 2024-06-06T08:30:06.780 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:06.780 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:06.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.782+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd60065430 msgr2=0x7fdd600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:06.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.782+0000 7fdd7a8ec640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd60065430 0x7fdd600678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdd740656e0 tx=0x7fdd5c009040 comp rx=0 tx=0).stop 2024-06-06T08:30:06.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.782+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 msgr2=0x7fdd7410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:06.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.782+0000 7fdd7a8ec640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 0x7fdd7410dfe0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdd64003080 tx=0x7fdd64009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:06.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.782+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 shutdown_connections 2024-06-06T08:30:06.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.782+0000 7fdd7a8ec640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd740fbe80 0x7fdd7410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:06.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.782+0000 7fdd7a8ec640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fdd60065430 0x7fdd600678e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:06.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.782+0000 7fdd7a8ec640 1 --2- 172.21.15.86:0/3487240457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd740fb570 0x7fdd7410dfe0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:06.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.782+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 >> 172.21.15.86:0/3487240457 conn(0x7fdd74072110 msgr2=0x7fdd740f86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:06.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.783+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 shutdown_connections 2024-06-06T08:30:06.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:06.783+0000 7fdd7a8ec640 1 -- 172.21.15.86:0/3487240457 wait complete. 2024-06-06T08:30:06.917 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:06 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3487240457' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:07.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:06 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3487240457' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:08.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:07 smithi086 ceph-mon[29114]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:08.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:07 smithi154 ceph-mon[35428]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:10.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:09 smithi086 ceph-mon[29114]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:10.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:30:10.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:09 smithi154 ceph-mon[35428]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:10.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:30:10.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:10.709 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:12.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:11 smithi086 ceph-mon[29114]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:12.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:11 smithi154 ceph-mon[35428]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.237+0000 7fa2412fe640 1 -- 172.21.15.86:0/71270024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 msgr2=0x7fa23c106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.237+0000 7fa2412fe640 1 --2- 172.21.15.86:0/71270024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 0x7fa23c106390 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa2300048d0 tx=0x7fa230009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.237+0000 7fa2412fe640 1 -- 172.21.15.86:0/71270024 shutdown_connections 2024-06-06T08:30:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.237+0000 7fa2412fe640 1 --2- 172.21.15.86:0/71270024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 0x7fa23c106390 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.237+0000 7fa2412fe640 1 --2- 172.21.15.86:0/71270024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa23c0fd320 0x7fa23c0fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.237+0000 7fa2412fe640 1 -- 172.21.15.86:0/71270024 >> 172.21.15.86:0/71270024 conn(0x7fa23c0f6dc0 msgr2=0x7fa23c0f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.237+0000 7fa2412fe640 1 -- 172.21.15.86:0/71270024 shutdown_connections 2024-06-06T08:30:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.237+0000 7fa2412fe640 1 -- 172.21.15.86:0/71270024 wait complete. 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa2412fe640 1 Processor -- start 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa2412fe640 1 -- start start 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa2412fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa23c0fd320 0x7fa23c1130b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa2412fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 0x7fa23c1135f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa2412fe640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c113bf0 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa2412fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c10d030 con 0x7fa23c0fd320 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa23a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 0x7fa23c1135f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa23a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 0x7fa23c1135f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44466/0 (socket says 172.21.15.86:44466) 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa23a7fc640 1 -- 172.21.15.86:0/1570335657 learned_addr learned my addr 172.21.15.86:0/1570335657 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa23affd640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa23c0fd320 0x7fa23c1130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa23a7fc640 1 -- 172.21.15.86:0/1570335657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa23c0fd320 msgr2=0x7fa23c1130b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa23a7fc640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa23c0fd320 0x7fa23c1130b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.238+0000 7fa23a7fc640 1 -- 172.21.15.86:0/1570335657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa230005040 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.239+0000 7fa23a7fc640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 0x7fa23c1135f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa2300048a0 tx=0x7fa230009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.239+0000 7fa2237fe640 1 -- 172.21.15.86:0/1570335657 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa230038070 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.239+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa23c10d2b0 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.239+0000 7fa2237fe640 1 -- 172.21.15.86:0/1570335657 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa23002abd0 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.239+0000 7fa2237fe640 1 -- 172.21.15.86:0/1570335657 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa230033770 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.239+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23c10d770 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.240+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23c065ff0 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.240+0000 7fa2237fe640 1 -- 172.21.15.86:0/1570335657 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa230030020 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.241+0000 7fa2237fe640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa224065430 0x7fa2240678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.241+0000 7fa23affd640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa224065430 0x7fa2240678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.241+0000 7fa2237fe640 1 -- 172.21.15.86:0/1570335657 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa2300a21b0 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.241+0000 7fa23affd640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa224065430 0x7fa2240678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa22c004ed0 tx=0x7fa22c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.245+0000 7fa2237fe640 1 -- 172.21.15.86:0/1570335657 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa230070710 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.442+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa23c0fa9c0 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.442+0000 7fa2237fe640 1 -- 172.21.15.86:0/1570335657 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa230073d30 con 0x7fa23c0fdcc0 2024-06-06T08:30:12.442 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:12.442 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa224065430 msgr2=0x7fa2240678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa224065430 0x7fa2240678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa22c004ed0 tx=0x7fa22c00b040 comp rx=0 tx=0).stop 2024-06-06T08:30:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 msgr2=0x7fa23c1135f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 0x7fa23c1135f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa2300048a0 tx=0x7fa230009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 shutdown_connections 2024-06-06T08:30:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fa224065430 0x7fa2240678e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa23c0fdcc0 0x7fa23c1135f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 --2- 172.21.15.86:0/1570335657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa23c0fd320 0x7fa23c1130b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 >> 172.21.15.86:0/1570335657 conn(0x7fa23c0f6dc0 msgr2=0x7fa23c100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 shutdown_connections 2024-06-06T08:30:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:12.444+0000 7fa2412fe640 1 -- 172.21.15.86:0/1570335657 wait complete. 2024-06-06T08:30:13.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:12 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1570335657' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:13.322 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:12 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1570335657' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:14.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:13 smithi154 ceph-mon[35428]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:14.257 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:13 smithi086 ceph-mon[29114]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:16.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:16.094 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:15 smithi086 ceph-mon[29114]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:16.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:15 smithi154 ceph-mon[35428]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:17.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.776+0000 7f27d1bc6640 1 -- 172.21.15.86:0/2073345255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 msgr2=0x7f27cc0fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:17.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.776+0000 7f27d1bc6640 1 --2- 172.21.15.86:0/2073345255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 0x7f27cc0fc2d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0048d0 tx=0x7f27bc009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:17.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.776+0000 7f27d1bc6640 1 -- 172.21.15.86:0/2073345255 shutdown_connections 2024-06-06T08:30:17.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.776+0000 7f27d1bc6640 1 --2- 172.21.15.86:0/2073345255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 0x7f27cc0fc2d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:17.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.776+0000 7f27d1bc6640 1 --2- 172.21.15.86:0/2073345255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc1022a0 0x7f27cc102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:17.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.776+0000 7f27d1bc6640 1 -- 172.21.15.86:0/2073345255 >> 172.21.15.86:0/2073345255 conn(0x7f27cc0f7e70 msgr2=0x7f27cc0fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:17.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.776+0000 7f27d1bc6640 1 -- 172.21.15.86:0/2073345255 shutdown_connections 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.776+0000 7f27d1bc6640 1 -- 172.21.15.86:0/2073345255 wait complete. 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.776+0000 7f27d1bc6640 1 Processor -- start 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27d1bc6640 1 -- start start 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27d1bc6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc1022a0 0x7f27cc1130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27d1bc6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 0x7f27cc113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27d1bc6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc113c90 con 0x7f27cc102c40 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27d1bc6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc10d040 con 0x7f27cc1022a0 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27caffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 0x7f27cc113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27caffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 0x7f27cc113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37918/0 (socket says 172.21.15.86:37918) 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27caffd640 1 -- 172.21.15.86:0/3527123915 learned_addr learned my addr 172.21.15.86:0/3527123915 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27cb7fe640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc1022a0 0x7f27cc1130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27caffd640 1 -- 172.21.15.86:0/3527123915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc1022a0 msgr2=0x7f27cc1130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27caffd640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc1022a0 0x7f27cc1130c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27caffd640 1 -- 172.21.15.86:0/3527123915 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc005040 con 0x7f27cc102c40 2024-06-06T08:30:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27cb7fe640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc1022a0 0x7f27cc1130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:30:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.777+0000 7f27caffd640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 0x7f27cc113600 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f27bc032430 tx=0x7f27bc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.778+0000 7f27b3fff640 1 -- 172.21.15.86:0/3527123915 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27bc038070 con 0x7f27cc102c40 2024-06-06T08:30:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.778+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc10d2c0 con 0x7f27cc102c40 2024-06-06T08:30:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.778+0000 7f27b3fff640 1 -- 172.21.15.86:0/3527123915 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27bc008ee0 con 0x7f27cc102c40 2024-06-06T08:30:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.778+0000 7f27b3fff640 1 -- 172.21.15.86:0/3527123915 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27bc03d7a0 con 0x7f27cc102c40 2024-06-06T08:30:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.778+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc10d780 con 0x7f27cc102c40 2024-06-06T08:30:17.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.779+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc0fda10 con 0x7f27cc102c40 2024-06-06T08:30:17.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.779+0000 7f27b3fff640 1 -- 172.21.15.86:0/3527123915 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f27bc030020 con 0x7f27cc102c40 2024-06-06T08:30:17.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.779+0000 7f27b3fff640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f27b4065430 0x7f27b40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:17.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.780+0000 7f27b3fff640 1 -- 172.21.15.86:0/3527123915 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f27bc0a2b20 con 0x7f27cc102c40 2024-06-06T08:30:17.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.780+0000 7f27cb7fe640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f27b4065430 0x7f27b40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:17.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.780+0000 7f27cb7fe640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f27b4065430 0x7f27b40678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f27c0006fd0 tx=0x7f27c0008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:17.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.784+0000 7f27b3fff640 1 -- 172.21.15.86:0/3527123915 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f27bc071090 con 0x7f27cc102c40 2024-06-06T08:30:17.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.982+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f27cc0fba90 con 0x7f27cc102c40 2024-06-06T08:30:17.981 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:17 smithi086 ceph-mon[29114]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:17.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.982+0000 7f27b3fff640 1 -- 172.21.15.86:0/3527123915 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f27cc0fba90 con 0x7f27cc102c40 2024-06-06T08:30:17.981 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:17.982 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:17.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f27b4065430 msgr2=0x7f27b40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:17.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f27b4065430 0x7f27b40678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f27c0006fd0 tx=0x7f27c0008040 comp rx=0 tx=0).stop 2024-06-06T08:30:17.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 msgr2=0x7f27cc113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:17.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 0x7f27cc113600 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f27bc032430 tx=0x7f27bc009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:17.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 shutdown_connections 2024-06-06T08:30:17.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f27b4065430 0x7f27b40678e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:17.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27cc102c40 0x7f27cc113600 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:17.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 --2- 172.21.15.86:0/3527123915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc1022a0 0x7f27cc1130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:17.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 >> 172.21.15.86:0/3527123915 conn(0x7f27cc0f7e70 msgr2=0x7f27cc0f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:17.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 shutdown_connections 2024-06-06T08:30:17.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:17.984+0000 7f27d1bc6640 1 -- 172.21.15.86:0/3527123915 wait complete. 2024-06-06T08:30:18.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:17 smithi154 ceph-mon[35428]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:19.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:18 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3527123915' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:19.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:18 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3527123915' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:20.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:19 smithi154 ceph-mon[35428]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:20.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:19 smithi086 ceph-mon[29114]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:21.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:21.962 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:21 smithi086 ceph-mon[29114]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:22.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:21 smithi154 ceph-mon[35428]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:23.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.264+0000 7fd16164f640 1 -- 172.21.15.86:0/3173667564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c101b80 msgr2=0x7fd15c0fb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:23.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.264+0000 7fd16164f640 1 --2- 172.21.15.86:0/3173667564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c101b80 0x7fd15c0fb210 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd1440048d0 tx=0x7fd144009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:23.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.264+0000 7fd16164f640 1 -- 172.21.15.86:0/3173667564 shutdown_connections 2024-06-06T08:30:23.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.264+0000 7fd16164f640 1 --2- 172.21.15.86:0/3173667564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c101b80 0x7fd15c0fb210 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:23.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.264+0000 7fd16164f640 1 --2- 172.21.15.86:0/3173667564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd15c1011e0 0x7fd15c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:23.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.264+0000 7fd16164f640 1 -- 172.21.15.86:0/3173667564 >> 172.21.15.86:0/3173667564 conn(0x7fd15c0f6d90 msgr2=0x7fd15c0f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:23.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.264+0000 7fd16164f640 1 -- 172.21.15.86:0/3173667564 shutdown_connections 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.264+0000 7fd16164f640 1 -- 172.21.15.86:0/3173667564 wait complete. 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd16164f640 1 Processor -- start 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd16164f640 1 -- start start 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd16164f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c1011e0 0x7fd15c10e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd16164f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd15c101b80 0x7fd15c10e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd16164f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c10ebb0 con 0x7fd15c1011e0 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd16164f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c10ed20 con 0x7fd15c101b80 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd15affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c1011e0 0x7fd15c10e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd15affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c1011e0 0x7fd15c10e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37938/0 (socket says 172.21.15.86:37938) 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd15affd640 1 -- 172.21.15.86:0/1734131901 learned_addr learned my addr 172.21.15.86:0/1734131901 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd15affd640 1 -- 172.21.15.86:0/1734131901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd15c101b80 msgr2=0x7fd15c10e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd15affd640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd15c101b80 0x7fd15c10e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:23.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd15affd640 1 -- 172.21.15.86:0/1734131901 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144005040 con 0x7fd15c1011e0 2024-06-06T08:30:23.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.265+0000 7fd15affd640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c1011e0 0x7fd15c10e020 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd15400db00 tx=0x7fd154004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.266+0000 7fd1537fe640 1 -- 172.21.15.86:0/1734131901 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd154017070 con 0x7fd15c1011e0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.266+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c117090 con 0x7fd15c1011e0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.266+0000 7fd1537fe640 1 -- 172.21.15.86:0/1734131901 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd15400f040 con 0x7fd15c1011e0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.266+0000 7fd1537fe640 1 -- 172.21.15.86:0/1734131901 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd154003ae0 con 0x7fd15c1011e0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.266+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c1175a0 con 0x7fd15c1011e0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.267+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c0fc950 con 0x7fd15c1011e0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.267+0000 7fd1537fe640 1 -- 172.21.15.86:0/1734131901 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd154003c40 con 0x7fd15c1011e0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.267+0000 7fd1537fe640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd148065430 0x7fd1480678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.267+0000 7fd1537fe640 1 -- 172.21.15.86:0/1734131901 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd1540825b0 con 0x7fd15c1011e0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.267+0000 7fd15a7fc640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd148065430 0x7fd1480678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:23.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.268+0000 7fd15a7fc640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd148065430 0x7fd1480678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd144032430 tx=0x7fd144009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:23.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.271+0000 7fd1537fe640 1 -- 172.21.15.86:0/1734131901 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd154050a60 con 0x7fd15c1011e0 2024-06-06T08:30:23.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.470+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd15c0fa9d0 con 0x7fd15c1011e0 2024-06-06T08:30:23.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.471+0000 7fd1537fe640 1 -- 172.21.15.86:0/1734131901 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd154054080 con 0x7fd15c1011e0 2024-06-06T08:30:23.470 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:23.470 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd148065430 msgr2=0x7fd1480678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd148065430 0x7fd1480678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd144032430 tx=0x7fd144009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c1011e0 msgr2=0x7fd15c10e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c1011e0 0x7fd15c10e020 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd15400db00 tx=0x7fd154004e60 comp rx=0 tx=0).stop 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 shutdown_connections 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd15c101b80 0x7fd15c10e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd148065430 0x7fd1480678e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 --2- 172.21.15.86:0/1734131901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd15c1011e0 0x7fd15c10e020 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 >> 172.21.15.86:0/1734131901 conn(0x7fd15c0f6d90 msgr2=0x7fd15c0f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 shutdown_connections 2024-06-06T08:30:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:23.473+0000 7fd16164f640 1 -- 172.21.15.86:0/1734131901 wait complete. 2024-06-06T08:30:24.004 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:23 smithi086 ceph-mon[29114]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:24.005 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:23 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1734131901' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:24.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:23 smithi154 ceph-mon[35428]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:24.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:23 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1734131901' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:25.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:24 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:30:25.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:30:26.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:25 smithi154 ceph-mon[35428]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:26.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:25 smithi086 ceph-mon[29114]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:27.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:27.272 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:27.964 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:27 smithi086 ceph-mon[29114]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:28.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:27 smithi154 ceph-mon[35428]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:28.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.702+0000 7f374660d640 1 -- 172.21.15.86:0/3051034789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 msgr2=0x7f37400fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:28.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.702+0000 7f374660d640 1 --2- 172.21.15.86:0/3051034789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 0x7f37400fc2f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f37340048d0 tx=0x7f3734009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.702+0000 7f374660d640 1 -- 172.21.15.86:0/3051034789 shutdown_connections 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.702+0000 7f374660d640 1 --2- 172.21.15.86:0/3051034789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 0x7f37400fc2f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.702+0000 7f374660d640 1 --2- 172.21.15.86:0/3051034789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37401022c0 0x7f3740102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.702+0000 7f374660d640 1 -- 172.21.15.86:0/3051034789 >> 172.21.15.86:0/3051034789 conn(0x7f37400f7e50 msgr2=0x7f37400fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.702+0000 7f374660d640 1 -- 172.21.15.86:0/3051034789 shutdown_connections 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.702+0000 7f374660d640 1 -- 172.21.15.86:0/3051034789 wait complete. 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f374660d640 1 Processor -- start 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f374660d640 1 -- start start 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f374660d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37401022c0 0x7f3740113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f374660d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 0x7f37401138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f374660d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3740113eb0 con 0x7f3740102c60 2024-06-06T08:30:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f374660d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374010d2f0 con 0x7f37401022c0 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f373f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 0x7f37401138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f373f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 0x7f37401138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39882/0 (socket says 172.21.15.86:39882) 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f373f7fe640 1 -- 172.21.15.86:0/979850093 learned_addr learned my addr 172.21.15.86:0/979850093 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f373ffff640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37401022c0 0x7f3740113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f373f7fe640 1 -- 172.21.15.86:0/979850093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37401022c0 msgr2=0x7f3740113370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f373f7fe640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37401022c0 0x7f3740113370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.703+0000 7f373f7fe640 1 -- 172.21.15.86:0/979850093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3734005040 con 0x7f3740102c60 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.704+0000 7f373f7fe640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 0x7f37401138b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3734003080 tx=0x7f3734009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.704+0000 7f373cff9640 1 -- 172.21.15.86:0/979850093 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3734038070 con 0x7f3740102c60 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.704+0000 7f373cff9640 1 -- 172.21.15.86:0/979850093 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3734032970 con 0x7f3740102c60 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.704+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374010d570 con 0x7f3740102c60 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.704+0000 7f373cff9640 1 -- 172.21.15.86:0/979850093 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f373403d960 con 0x7f3740102c60 2024-06-06T08:30:28.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.704+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374010dab0 con 0x7f3740102c60 2024-06-06T08:30:28.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.705+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37400fda30 con 0x7f3740102c60 2024-06-06T08:30:28.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.705+0000 7f373cff9640 1 -- 172.21.15.86:0/979850093 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f3734030020 con 0x7f3740102c60 2024-06-06T08:30:28.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.706+0000 7f373cff9640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3728065430 0x7f37280678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:28.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.706+0000 7f373cff9640 1 -- 172.21.15.86:0/979850093 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f37340a2480 con 0x7f3740102c60 2024-06-06T08:30:28.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.706+0000 7f373ffff640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3728065430 0x7f37280678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:28.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.706+0000 7f373ffff640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3728065430 0x7f37280678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3730009af0 tx=0x7f373000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:28.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.710+0000 7f373cff9640 1 -- 172.21.15.86:0/979850093 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3734070930 con 0x7f3740102c60 2024-06-06T08:30:28.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.912+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3740109420 con 0x7f3740102c60 2024-06-06T08:30:28.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.912+0000 7f373cff9640 1 -- 172.21.15.86:0/979850093 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3734073f50 con 0x7f3740102c60 2024-06-06T08:30:28.912 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:28.912 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:28.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.914+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3728065430 msgr2=0x7f37280678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:28.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.914+0000 7f374660d640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3728065430 0x7f37280678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3730009af0 tx=0x7f373000b040 comp rx=0 tx=0).stop 2024-06-06T08:30:28.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.914+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 msgr2=0x7f37401138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:28.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.914+0000 7f374660d640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 0x7f37401138b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3734003080 tx=0x7f3734009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:28.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.914+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 shutdown_connections 2024-06-06T08:30:28.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.914+0000 7f374660d640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f3728065430 0x7f37280678e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:28.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.914+0000 7f374660d640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3740102c60 0x7f37401138b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:28.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.914+0000 7f374660d640 1 --2- 172.21.15.86:0/979850093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37401022c0 0x7f3740113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:28.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.914+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 >> 172.21.15.86:0/979850093 conn(0x7f37400f7e50 msgr2=0x7f37400f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:28.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.915+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 shutdown_connections 2024-06-06T08:30:28.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:28.915+0000 7f374660d640 1 -- 172.21.15.86:0/979850093 wait complete. 2024-06-06T08:30:30.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:29 smithi154 ceph-mon[35428]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:30.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:29 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/979850093' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:30.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:29 smithi086 ceph-mon[29114]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:30.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:29 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/979850093' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:32.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:31 smithi154 ceph-mon[35428]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:32.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:31 smithi086 ceph-mon[29114]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:32.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:32.882 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:34.149 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:33 smithi086 ceph-mon[29114]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:34.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:33 smithi154 ceph-mon[35428]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:34.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 -- 172.21.15.86:0/3791676770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 msgr2=0x7f90240fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:34.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 --2- 172.21.15.86:0/3791676770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 0x7f90240fb940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f901c009c50 tx=0x7f901c004d10 comp rx=0 tx=0).stop 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 -- 172.21.15.86:0/3791676770 shutdown_connections 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 --2- 172.21.15.86:0/3791676770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90240fbe80 0x7f90240fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 --2- 172.21.15.86:0/3791676770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 0x7f90240fb940 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 -- 172.21.15.86:0/3791676770 >> 172.21.15.86:0/3791676770 conn(0x7f9024072110 msgr2=0x7f9024072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 -- 172.21.15.86:0/3791676770 shutdown_connections 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 -- 172.21.15.86:0/3791676770 wait complete. 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 Processor -- start 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.330+0000 7f9028a8e640 1 -- start start 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f9028a8e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 0x7f9024113010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:34.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f9028a8e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90240fbe80 0x7f9024113550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f9028a8e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9024113b50 con 0x7f90240fb570 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f9028a8e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902410cf90 con 0x7f90240fbe80 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f902259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 0x7f9024113010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f902259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 0x7f9024113010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39890/0 (socket says 172.21.15.86:39890) 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f902259c640 1 -- 172.21.15.86:0/1678748109 learned_addr learned my addr 172.21.15.86:0/1678748109 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f9021d9b640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90240fbe80 0x7f9024113550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f902259c640 1 -- 172.21.15.86:0/1678748109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90240fbe80 msgr2=0x7f9024113550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f902259c640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90240fbe80 0x7f9024113550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f902259c640 1 -- 172.21.15.86:0/1678748109 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c0098b0 con 0x7f90240fb570 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f902259c640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 0x7f9024113010 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f901c004f80 tx=0x7f901c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f901affd640 1 -- 172.21.15.86:0/1678748109 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f901c008de0 con 0x7f90240fb570 2024-06-06T08:30:34.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902410d210 con 0x7f90240fb570 2024-06-06T08:30:34.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f901affd640 1 -- 172.21.15.86:0/1678748109 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f901c032c00 con 0x7f90240fb570 2024-06-06T08:30:34.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f901affd640 1 -- 172.21.15.86:0/1678748109 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f901c03dbc0 con 0x7f90240fb570 2024-06-06T08:30:34.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.331+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902410d730 con 0x7f90240fb570 2024-06-06T08:30:34.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.332+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90240fc920 con 0x7f90240fb570 2024-06-06T08:30:34.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.333+0000 7f901affd640 1 -- 172.21.15.86:0/1678748109 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f901c02f030 con 0x7f90240fb570 2024-06-06T08:30:34.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.333+0000 7f901affd640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9010065430 0x7f90100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:34.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.333+0000 7f9021d9b640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9010065430 0x7f90100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:34.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.333+0000 7f901affd640 1 -- 172.21.15.86:0/1678748109 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f901c0a3ba0 con 0x7f90240fb570 2024-06-06T08:30:34.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.333+0000 7f9021d9b640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9010065430 0x7f90100678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f902410e8d0 tx=0x7f900c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:34.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.337+0000 7f901affd640 1 -- 172.21.15.86:0/1678748109 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f901c072050 con 0x7f90240fb570 2024-06-06T08:30:34.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.535+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f902404b480 con 0x7f90240fb570 2024-06-06T08:30:34.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.535+0000 7f901affd640 1 -- 172.21.15.86:0/1678748109 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f901c075670 con 0x7f90240fb570 2024-06-06T08:30:34.534 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:34.534 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9010065430 msgr2=0x7f90100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9010065430 0x7f90100678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f902410e8d0 tx=0x7f900c009040 comp rx=0 tx=0).stop 2024-06-06T08:30:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 msgr2=0x7f9024113010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 0x7f9024113010 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f901c004f80 tx=0x7f901c004d10 comp rx=0 tx=0).stop 2024-06-06T08:30:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 shutdown_connections 2024-06-06T08:30:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90240fbe80 0x7f9024113550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f9010065430 0x7f90100678e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 --2- 172.21.15.86:0/1678748109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90240fb570 0x7f9024113010 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 >> 172.21.15.86:0/1678748109 conn(0x7f9024072110 msgr2=0x7f90240ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:34.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 shutdown_connections 2024-06-06T08:30:34.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:34.537+0000 7f9028a8e640 1 -- 172.21.15.86:0/1678748109 wait complete. 2024-06-06T08:30:35.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:34 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1678748109' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:35.314 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:34 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1678748109' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:36.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:35 smithi154 ceph-mon[35428]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:36.237 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:35 smithi086 ceph-mon[29114]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:38.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:38.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:37 smithi154 ceph-mon[35428]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:38.323 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:38.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:37 smithi086 ceph-mon[29114]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:39.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.720+0000 7fae0b920640 1 -- 172.21.15.86:0/4269653608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fb570 msgr2=0x7fae040fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:39.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.720+0000 7fae0b920640 1 --2- 172.21.15.86:0/4269653608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fb570 0x7fae040fb940 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fae000048d0 tx=0x7fae00009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 -- 172.21.15.86:0/4269653608 shutdown_connections 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 --2- 172.21.15.86:0/4269653608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae040fbe80 0x7fae040fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 --2- 172.21.15.86:0/4269653608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fb570 0x7fae040fb940 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 -- 172.21.15.86:0/4269653608 >> 172.21.15.86:0/4269653608 conn(0x7fae04072110 msgr2=0x7fae04072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 -- 172.21.15.86:0/4269653608 shutdown_connections 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 -- 172.21.15.86:0/4269653608 wait complete. 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 Processor -- start 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 -- start start 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae040fb570 0x7fae040f8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fbe80 0x7fae040f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae040f9130 con 0x7fae040fbe80 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae0b920640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae040f92a0 con 0x7fae040fb570 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.721+0000 7fae08ebb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fbe80 0x7fae040f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:39.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fae08ebb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fbe80 0x7fae040f8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51626/0 (socket says 172.21.15.86:51626) 2024-06-06T08:30:39.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fae08ebb640 1 -- 172.21.15.86:0/2531717870 learned_addr learned my addr 172.21.15.86:0/2531717870 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:39.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fae096bc640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae040fb570 0x7fae040f8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:39.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fae08ebb640 1 -- 172.21.15.86:0/2531717870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae040fb570 msgr2=0x7fae040f8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:39.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fae08ebb640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae040fb570 0x7fae040f8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:39.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fae08ebb640 1 -- 172.21.15.86:0/2531717870 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00005040 con 0x7fae040fbe80 2024-06-06T08:30:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fae08ebb640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fbe80 0x7fae040f8b60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fadf400b790 tx=0x7fadf40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fadf9ffb640 1 -- 172.21.15.86:0/2531717870 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf400c700 con 0x7fae040fbe80 2024-06-06T08:30:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fadf9ffb640 1 -- 172.21.15.86:0/2531717870 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadf400cd30 con 0x7fae040fbe80 2024-06-06T08:30:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fadf9ffb640 1 -- 172.21.15.86:0/2531717870 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf4013370 con 0x7fae040fbe80 2024-06-06T08:30:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae041193a0 con 0x7fae040fbe80 2024-06-06T08:30:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.722+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae041197e0 con 0x7fae040fbe80 2024-06-06T08:30:39.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.724+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae040fc920 con 0x7fae040fbe80 2024-06-06T08:30:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.724+0000 7fadf9ffb640 1 -- 172.21.15.86:0/2531717870 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fadf4012070 con 0x7fae040fbe80 2024-06-06T08:30:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.724+0000 7fadf9ffb640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fadf0065430 0x7fadf00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.724+0000 7fadf9ffb640 1 -- 172.21.15.86:0/2531717870 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fadf4081dd0 con 0x7fae040fbe80 2024-06-06T08:30:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.724+0000 7fae096bc640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fadf0065430 0x7fadf00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:39.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.725+0000 7fae096bc640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fadf0065430 0x7fadf00678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fae00009f80 tx=0x7fae00035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:39.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.728+0000 7fadf9ffb640 1 -- 172.21.15.86:0/2531717870 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fadf4050280 con 0x7fae040fbe80 2024-06-06T08:30:39.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.928+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fae04072c20 con 0x7fae040fbe80 2024-06-06T08:30:39.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.929+0000 7fadf9ffb640 1 -- 172.21.15.86:0/2531717870 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fadf40538a0 con 0x7fae040fbe80 2024-06-06T08:30:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:39.928 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:39.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fadf0065430 msgr2=0x7fadf00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:39.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fadf0065430 0x7fadf00678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fae00009f80 tx=0x7fae00035040 comp rx=0 tx=0).stop 2024-06-06T08:30:39.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fbe80 msgr2=0x7fae040f8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:39.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fbe80 0x7fae040f8b60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fadf400b790 tx=0x7fadf40091d0 comp rx=0 tx=0).stop 2024-06-06T08:30:39.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 shutdown_connections 2024-06-06T08:30:39.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fadf0065430 0x7fadf00678e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:39.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae040fbe80 0x7fae040f8b60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:39.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 --2- 172.21.15.86:0/2531717870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae040fb570 0x7fae040f8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 >> 172.21.15.86:0/2531717870 conn(0x7fae04072110 msgr2=0x7fae040ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 shutdown_connections 2024-06-06T08:30:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:39.931+0000 7fae0b920640 1 -- 172.21.15.86:0/2531717870 wait complete. 2024-06-06T08:30:40.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:39 smithi154 ceph-mon[35428]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:40.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:39 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:30:40.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:39 smithi086 ceph-mon[29114]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:40.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:39 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:30:41.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:40 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2531717870' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:41.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:40 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2531717870' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:42.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:41 smithi154 ceph-mon[35428]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:42.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:41 smithi086 ceph-mon[29114]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:43.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:43.734 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:44.122 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:43 smithi086 ceph-mon[29114]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:44.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:43 smithi154 ceph-mon[35428]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.164+0000 7fbfecd14640 1 -- 172.21.15.86:0/776512035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe8102c60 msgr2=0x7fbfe80fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.164+0000 7fbfecd14640 1 --2- 172.21.15.86:0/776512035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe8102c60 0x7fbfe80fc2f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00048d0 tx=0x7fbfd0009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.164+0000 7fbfecd14640 1 -- 172.21.15.86:0/776512035 shutdown_connections 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.164+0000 7fbfecd14640 1 --2- 172.21.15.86:0/776512035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe8102c60 0x7fbfe80fc2f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.164+0000 7fbfecd14640 1 --2- 172.21.15.86:0/776512035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfe81022c0 0x7fbfe8102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.164+0000 7fbfecd14640 1 -- 172.21.15.86:0/776512035 >> 172.21.15.86:0/776512035 conn(0x7fbfe80f7e50 msgr2=0x7fbfe80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.164+0000 7fbfecd14640 1 -- 172.21.15.86:0/776512035 shutdown_connections 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.164+0000 7fbfecd14640 1 -- 172.21.15.86:0/776512035 wait complete. 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.165+0000 7fbfecd14640 1 Processor -- start 2024-06-06T08:30:45.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.165+0000 7fbfecd14640 1 -- start start 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.165+0000 7fbfecd14640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe81022c0 0x7fbfe8113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.165+0000 7fbfecd14640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfe8102c60 0x7fbfe8113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.165+0000 7fbfecd14640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe8113e90 con 0x7fbfe81022c0 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.165+0000 7fbfecd14640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe810d2d0 con 0x7fbfe8102c60 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.165+0000 7fbfe659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe81022c0 0x7fbfe8113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.165+0000 7fbfe659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe81022c0 0x7fbfe8113350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51644/0 (socket says 172.21.15.86:51644) 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.165+0000 7fbfe659c640 1 -- 172.21.15.86:0/1800541993 learned_addr learned my addr 172.21.15.86:0/1800541993 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfdffff640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfe8102c60 0x7fbfe8113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfe659c640 1 -- 172.21.15.86:0/1800541993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfe8102c60 msgr2=0x7fbfe8113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfe659c640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfe8102c60 0x7fbfe8113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfe659c640 1 -- 172.21.15.86:0/1800541993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd0005040 con 0x7fbfe81022c0 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfe659c640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe81022c0 0x7fbfe8113350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800c830 tx=0x7fbfd8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfdeffd640 1 -- 172.21.15.86:0/1800541993 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd800a7d0 con 0x7fbfe81022c0 2024-06-06T08:30:45.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfdeffd640 1 -- 172.21.15.86:0/1800541993 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfd800ae00 con 0x7fbfe81022c0 2024-06-06T08:30:45.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfdeffd640 1 -- 172.21.15.86:0/1800541993 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd80113a0 con 0x7fbfe81022c0 2024-06-06T08:30:45.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe810d5b0 con 0x7fbfe81022c0 2024-06-06T08:30:45.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.166+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe810db50 con 0x7fbfe81022c0 2024-06-06T08:30:45.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.167+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe80fda30 con 0x7fbfe81022c0 2024-06-06T08:30:45.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.167+0000 7fbfdeffd640 1 -- 172.21.15.86:0/1800541993 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fbfd8012400 con 0x7fbfe81022c0 2024-06-06T08:30:45.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.168+0000 7fbfdeffd640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbfd4065430 0x7fbfd40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:45.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.168+0000 7fbfdffff640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbfd4065430 0x7fbfd40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:45.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.168+0000 7fbfdeffd640 1 -- 172.21.15.86:0/1800541993 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbfd80825d0 con 0x7fbfe81022c0 2024-06-06T08:30:45.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.168+0000 7fbfdffff640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbfd4065430 0x7fbfd40678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0032430 tx=0x7fbfd0009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:45.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.172+0000 7fbfdeffd640 1 -- 172.21.15.86:0/1800541993 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbfd8050ba0 con 0x7fbfe81022c0 2024-06-06T08:30:45.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.371+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbfe80fbab0 con 0x7fbfe81022c0 2024-06-06T08:30:45.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.371+0000 7fbfdeffd640 1 -- 172.21.15.86:0/1800541993 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbfd80541c0 con 0x7fbfe81022c0 2024-06-06T08:30:45.370 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:45.371 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:45.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbfd4065430 msgr2=0x7fbfd40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:45.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbfd4065430 0x7fbfd40678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0032430 tx=0x7fbfd0009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:45.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe81022c0 msgr2=0x7fbfe8113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:45.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe81022c0 0x7fbfe8113350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800c830 tx=0x7fbfd8004430 comp rx=0 tx=0).stop 2024-06-06T08:30:45.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 shutdown_connections 2024-06-06T08:30:45.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfe8102c60 0x7fbfe8113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:45.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fbfd4065430 0x7fbfd40678e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:45.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 --2- 172.21.15.86:0/1800541993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe81022c0 0x7fbfe8113350 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:45.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 >> 172.21.15.86:0/1800541993 conn(0x7fbfe80f7e50 msgr2=0x7fbfe80f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:45.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 shutdown_connections 2024-06-06T08:30:45.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:45.373+0000 7fbfecd14640 1 -- 172.21.15.86:0/1800541993 wait complete. 2024-06-06T08:30:45.937 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:45 smithi086 ceph-mon[29114]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:45.937 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:45 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1800541993' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:46.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:45 smithi154 ceph-mon[35428]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:46.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:45 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1800541993' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:48.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:47 smithi154 ceph-mon[35428]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:48.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:47 smithi086 ceph-mon[29114]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:48.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:49.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:48 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:30:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:49.317 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:48 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:30:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:49 smithi154 ceph-mon[35428]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:49 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:30:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:49 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:30:50.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:49 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:30:50.189 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:49 smithi086 ceph-mon[29114]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:50.189 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:49 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:30:50.189 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:49 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:30:50.189 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:49 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:30:50.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 -- 172.21.15.86:0/3314793094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 msgr2=0x7f62540faa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:50.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 --2- 172.21.15.86:0/3314793094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 0x7f62540faa50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6248009c50 tx=0x7f6248004d10 comp rx=0 tx=0).stop 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 -- 172.21.15.86:0/3314793094 shutdown_connections 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 --2- 172.21.15.86:0/3314793094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 0x7f62540faa50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 --2- 172.21.15.86:0/3314793094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254102380 0x7f6254102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 -- 172.21.15.86:0/3314793094 >> 172.21.15.86:0/3314793094 conn(0x7f6254070ab0 msgr2=0x7f6254070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 -- 172.21.15.86:0/3314793094 shutdown_connections 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 -- 172.21.15.86:0/3314793094 wait complete. 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 Processor -- start 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.467+0000 7f625c21a640 1 -- start start 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f625c21a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254102380 0x7f62540fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f625c21a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 0x7f62540fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f625c21a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62540fc6b0 con 0x7f6254102d20 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f625c21a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62540fc820 con 0x7f6254102380 2024-06-06T08:30:50.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f62597b5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 0x7f62540fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f62597b5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 0x7f62540fc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39362/0 (socket says 172.21.15.86:39362) 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f62597b5640 1 -- 172.21.15.86:0/3248077194 learned_addr learned my addr 172.21.15.86:0/3248077194 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f6259fb6640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254102380 0x7f62540fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f62597b5640 1 -- 172.21.15.86:0/3248077194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254102380 msgr2=0x7f62540fbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f62597b5640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254102380 0x7f62540fbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f62597b5640 1 -- 172.21.15.86:0/3248077194 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62480098b0 con 0x7f6254102d20 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f62597b5640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 0x7f62540fc170 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6248004750 tx=0x7f6248004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f62527fc640 1 -- 172.21.15.86:0/3248077194 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6248041070 con 0x7f6254102d20 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.468+0000 7f62527fc640 1 -- 172.21.15.86:0/3248077194 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6248007d50 con 0x7f6254102d20 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.469+0000 7f62527fc640 1 -- 172.21.15.86:0/3248077194 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f624803da00 con 0x7f6254102d20 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.469+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62540fca50 con 0x7f6254102d20 2024-06-06T08:30:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.469+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62541196b0 con 0x7f6254102d20 2024-06-06T08:30:50.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.469+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6254065520 con 0x7f6254102d20 2024-06-06T08:30:50.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.470+0000 7f62527fc640 1 -- 172.21.15.86:0/3248077194 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f624802f030 con 0x7f6254102d20 2024-06-06T08:30:50.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.470+0000 7f62527fc640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f623c065430 0x7f623c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:50.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.470+0000 7f6259fb6640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f623c065430 0x7f623c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:50.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.470+0000 7f62527fc640 1 -- 172.21.15.86:0/3248077194 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f62480a2e10 con 0x7f6254102d20 2024-06-06T08:30:50.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.471+0000 7f6259fb6640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f623c065430 0x7f623c0678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f624c004ed0 tx=0x7f624c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:50.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.475+0000 7f62527fc640 1 -- 172.21.15.86:0/3248077194 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6248071340 con 0x7f6254102d20 2024-06-06T08:30:50.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.671+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f62540749d0 con 0x7f6254102d20 2024-06-06T08:30:50.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.671+0000 7f62527fc640 1 -- 172.21.15.86:0/3248077194 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6248074960 con 0x7f6254102d20 2024-06-06T08:30:50.670 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:50.670 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:50.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f623c065430 msgr2=0x7f623c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:50.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f623c065430 0x7f623c0678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f624c004ed0 tx=0x7f624c00b040 comp rx=0 tx=0).stop 2024-06-06T08:30:50.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 msgr2=0x7f62540fc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:50.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 0x7f62540fc170 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6248004750 tx=0x7f6248004d10 comp rx=0 tx=0).stop 2024-06-06T08:30:50.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 shutdown_connections 2024-06-06T08:30:50.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f623c065430 0x7f623c0678e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:50.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254102d20 0x7f62540fc170 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:50.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 --2- 172.21.15.86:0/3248077194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254102380 0x7f62540fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:50.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 >> 172.21.15.86:0/3248077194 conn(0x7f6254070ab0 msgr2=0x7f62540fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:50.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 shutdown_connections 2024-06-06T08:30:50.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:50.673+0000 7f625c21a640 1 -- 172.21.15.86:0/3248077194 wait complete. 2024-06-06T08:30:51.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:50 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3248077194' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:51.169 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:50 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3248077194' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:52.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:51 smithi154 ceph-mon[35428]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:52.294 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:51 smithi086 ceph-mon[29114]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:54.166 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:53 smithi154 ceph-mon[35428]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:54.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:30:54.220 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:53 smithi086 ceph-mon[29114]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:30:55.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:54 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:30:55.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:54 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:30:55.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.873+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/2937889381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fd2a0 msgr2=0x7f0a480fd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:55.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.873+0000 7f0a4dcfc640 1 --2- 172.21.15.86:0/2937889381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fd2a0 0x7f0a480fd670 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0a38004900 tx=0x7f0a38009da0 comp rx=0 tx=0).stop 2024-06-06T08:30:55.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.873+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/2937889381 shutdown_connections 2024-06-06T08:30:55.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.873+0000 7f0a4dcfc640 1 --2- 172.21.15.86:0/2937889381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a480fdc40 0x7f0a48106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:55.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.873+0000 7f0a4dcfc640 1 --2- 172.21.15.86:0/2937889381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fd2a0 0x7f0a480fd670 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:55.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.873+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/2937889381 >> 172.21.15.86:0/2937889381 conn(0x7f0a480f6d80 msgr2=0x7f0a480f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:55.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.873+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/2937889381 shutdown_connections 2024-06-06T08:30:55.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.873+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/2937889381 wait complete. 2024-06-06T08:30:55.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a4dcfc640 1 Processor -- start 2024-06-06T08:30:55.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a4dcfc640 1 -- start start 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a4dcfc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a480fd2a0 0x7f0a48113110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a4dcfc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fdc40 0x7f0a48113650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a4dcfc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a48113ce0 con 0x7f0a480fdc40 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a4dcfc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4810d090 con 0x7f0a480fd2a0 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a3ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fdc40 0x7f0a48113650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a3ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fdc40 0x7f0a48113650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39372/0 (socket says 172.21.15.86:39372) 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a3ffff640 1 -- 172.21.15.86:0/604410232 learned_addr learned my addr 172.21.15.86:0/604410232 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a477fe640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a480fd2a0 0x7f0a48113110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a3ffff640 1 -- 172.21.15.86:0/604410232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a480fd2a0 msgr2=0x7f0a48113110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a3ffff640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a480fd2a0 0x7f0a48113110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.874+0000 7f0a3ffff640 1 -- 172.21.15.86:0/604410232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a38005040 con 0x7f0a480fdc40 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.875+0000 7f0a3ffff640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fdc40 0x7f0a48113650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000b790 tx=0x7f0a300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:55.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.875+0000 7f0a44ff9640 1 -- 172.21.15.86:0/604410232 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a3000c700 con 0x7f0a480fdc40 2024-06-06T08:30:55.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.875+0000 7f0a44ff9640 1 -- 172.21.15.86:0/604410232 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a3000cd30 con 0x7f0a480fdc40 2024-06-06T08:30:55.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.875+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4810d370 con 0x7f0a480fdc40 2024-06-06T08:30:55.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.875+0000 7f0a44ff9640 1 -- 172.21.15.86:0/604410232 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a30013370 con 0x7f0a480fdc40 2024-06-06T08:30:55.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.875+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4810d890 con 0x7f0a480fdc40 2024-06-06T08:30:55.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.876+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a48066070 con 0x7f0a480fdc40 2024-06-06T08:30:55.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.876+0000 7f0a44ff9640 1 -- 172.21.15.86:0/604410232 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0a30012070 con 0x7f0a480fdc40 2024-06-06T08:30:55.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.877+0000 7f0a44ff9640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0a34065430 0x7f0a340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:30:55.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.877+0000 7f0a477fe640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0a34065430 0x7f0a340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:30:55.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.877+0000 7f0a44ff9640 1 -- 172.21.15.86:0/604410232 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0a30081dd0 con 0x7f0a480fdc40 2024-06-06T08:30:55.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.877+0000 7f0a477fe640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0a34065430 0x7f0a340678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0a38000c00 tx=0x7f0a38035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:30:55.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:55.881+0000 7f0a44ff9640 1 -- 172.21.15.86:0/604410232 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0a30050280 con 0x7f0a480fdc40 2024-06-06T08:30:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.081+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0a481094c0 con 0x7f0a480fdc40 2024-06-06T08:30:56.081 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:55 smithi086 ceph-mon[29114]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.082+0000 7f0a44ff9640 1 -- 172.21.15.86:0/604410232 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0a300538a0 con 0x7f0a480fdc40 2024-06-06T08:30:56.081 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:30:56.082 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0a34065430 msgr2=0x7f0a340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0a34065430 0x7f0a340678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0a38000c00 tx=0x7f0a38035040 comp rx=0 tx=0).stop 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fdc40 msgr2=0x7f0a48113650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fdc40 0x7f0a48113650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000b790 tx=0x7f0a300091d0 comp rx=0 tx=0).stop 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 shutdown_connections 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0a34065430 0x7f0a340678e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a480fdc40 0x7f0a48113650 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 --2- 172.21.15.86:0/604410232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a480fd2a0 0x7f0a48113110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 >> 172.21.15.86:0/604410232 conn(0x7f0a480f6d80 msgr2=0x7f0a480f8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 shutdown_connections 2024-06-06T08:30:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:30:56.084+0000 7f0a4dcfc640 1 -- 172.21.15.86:0/604410232 wait complete. 2024-06-06T08:30:56.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:55 smithi154 ceph-mon[35428]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:57.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:56 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/604410232' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:57.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:56 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/604410232' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:30:58.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:57 smithi086 ceph-mon[29114]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:58.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:57 smithi154 ceph-mon[35428]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:30:59.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:00.018 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:00.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:30:59 smithi086 ceph-mon[29114]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:00.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:30:59 smithi154 ceph-mon[35428]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:01.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.470+0000 7f444f0fe640 1 -- 172.21.15.86:0/3656335002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4448101b60 msgr2=0x7f44480fb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:01.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.470+0000 7f444f0fe640 1 --2- 172.21.15.86:0/3656335002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4448101b60 0x7f44480fb1f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f44380048d0 tx=0x7f4438009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:01.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.470+0000 7f444f0fe640 1 -- 172.21.15.86:0/3656335002 shutdown_connections 2024-06-06T08:31:01.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.470+0000 7f444f0fe640 1 --2- 172.21.15.86:0/3656335002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4448101b60 0x7f44480fb1f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:01.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.470+0000 7f444f0fe640 1 --2- 172.21.15.86:0/3656335002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44481011c0 0x7f4448101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:01.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.470+0000 7f444f0fe640 1 -- 172.21.15.86:0/3656335002 >> 172.21.15.86:0/3656335002 conn(0x7f44480f6d70 msgr2=0x7f44480f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:01.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.470+0000 7f444f0fe640 1 -- 172.21.15.86:0/3656335002 shutdown_connections 2024-06-06T08:31:01.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.470+0000 7f444f0fe640 1 -- 172.21.15.86:0/3656335002 wait complete. 2024-06-06T08:31:01.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.471+0000 7f444f0fe640 1 Processor -- start 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.471+0000 7f444f0fe640 1 -- start start 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.471+0000 7f444f0fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44481011c0 0x7f444810df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.471+0000 7f444f0fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4448101b60 0x7f444810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.471+0000 7f444f0fe640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444810ead0 con 0x7f44481011c0 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.471+0000 7f444f0fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444810ec40 con 0x7f4448101b60 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.471+0000 7f444ce9a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44481011c0 0x7f444810df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.471+0000 7f444ce9a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44481011c0 0x7f444810df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60000/0 (socket says 172.21.15.86:60000) 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.471+0000 7f444ce9a640 1 -- 172.21.15.86:0/3614503266 learned_addr learned my addr 172.21.15.86:0/3614503266 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f443ffff640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4448101b60 0x7f444810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f444ce9a640 1 -- 172.21.15.86:0/3614503266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4448101b60 msgr2=0x7f444810e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f444ce9a640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4448101b60 0x7f444810e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f444ce9a640 1 -- 172.21.15.86:0/3614503266 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4438005040 con 0x7f44481011c0 2024-06-06T08:31:01.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f443ffff640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4448101b60 0x7f444810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f444ce9a640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44481011c0 0x7f444810df90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f444400c830 tx=0x7f4444004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f443d7fa640 1 -- 172.21.15.86:0/3614503266 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f444400a850 con 0x7f44481011c0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f443d7fa640 1 -- 172.21.15.86:0/3614503266 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f444400ae80 con 0x7f44481011c0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4448117000 con 0x7f44481011c0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f443d7fa640 1 -- 172.21.15.86:0/3614503266 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4444012ea0 con 0x7f44481011c0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.472+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4448117540 con 0x7f44481011c0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.473+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44480fc930 con 0x7f44481011c0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.473+0000 7f443d7fa640 1 -- 172.21.15.86:0/3614503266 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4444012400 con 0x7f44481011c0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.474+0000 7f443d7fa640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4430065430 0x7f44300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.474+0000 7f443d7fa640 1 -- 172.21.15.86:0/3614503266 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4444082730 con 0x7f44481011c0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.474+0000 7f443ffff640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4430065430 0x7f44300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:01.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.474+0000 7f443ffff640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4430065430 0x7f44300678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4438000c00 tx=0x7f4438035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:01.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.478+0000 7f443d7fa640 1 -- 172.21.15.86:0/3614503266 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4444050c60 con 0x7f44481011c0 2024-06-06T08:31:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.679+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f44480fa9b0 con 0x7f44481011c0 2024-06-06T08:31:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.679+0000 7f443d7fa640 1 -- 172.21.15.86:0/3614503266 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f44440126b0 con 0x7f44481011c0 2024-06-06T08:31:01.679 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4430065430 msgr2=0x7f44300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4430065430 0x7f44300678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4438000c00 tx=0x7f4438035040 comp rx=0 tx=0).stop 2024-06-06T08:31:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44481011c0 msgr2=0x7f444810df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44481011c0 0x7f444810df90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f444400c830 tx=0x7f4444004430 comp rx=0 tx=0).stop 2024-06-06T08:31:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 shutdown_connections 2024-06-06T08:31:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4448101b60 0x7f444810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:01.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4430065430 0x7f44300678e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 --2- 172.21.15.86:0/3614503266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44481011c0 0x7f444810df90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 >> 172.21.15.86:0/3614503266 conn(0x7f44480f6d70 msgr2=0x7f44480f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 shutdown_connections 2024-06-06T08:31:01.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:01.681+0000 7f444f0fe640 1 -- 172.21.15.86:0/3614503266 wait complete. 2024-06-06T08:31:02.290 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:01 smithi086 ceph-mon[29114]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:02.290 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:01 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3614503266' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:02.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:01 smithi154 ceph-mon[35428]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:02.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:01 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3614503266' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:04.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:03 smithi086 ceph-mon[29114]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:03 smithi154 ceph-mon[35428]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:05.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:05.976 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:05 smithi086 ceph-mon[29114]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:06.219 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:05 smithi154 ceph-mon[35428]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:06.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.942+0000 7f8aa43f4640 1 -- 172.21.15.86:0/3628841380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c0fa9c0 msgr2=0x7f8a9c0fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:06.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.942+0000 7f8aa43f4640 1 --2- 172.21.15.86:0/3628841380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c0fa9c0 0x7f8a9c0fae30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8a980048d0 tx=0x7f8a98009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:06.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.942+0000 7f8aa43f4640 1 -- 172.21.15.86:0/3628841380 shutdown_connections 2024-06-06T08:31:06.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.942+0000 7f8aa43f4640 1 --2- 172.21.15.86:0/3628841380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c0fa9c0 0x7f8a9c0fae30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:06.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.942+0000 7f8aa43f4640 1 --2- 172.21.15.86:0/3628841380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c102370 0x7f8a9c102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:06.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.942+0000 7f8aa43f4640 1 -- 172.21.15.86:0/3628841380 >> 172.21.15.86:0/3628841380 conn(0x7f8a9c0709e0 msgr2=0x7f8a9c070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.942+0000 7f8aa43f4640 1 -- 172.21.15.86:0/3628841380 shutdown_connections 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.942+0000 7f8aa43f4640 1 -- 172.21.15.86:0/3628841380 wait complete. 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa43f4640 1 Processor -- start 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa43f4640 1 -- start start 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa43f4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c0fa9c0 0x7f8a9c113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa43f4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c102370 0x7f8a9c113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa43f4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c113b40 con 0x7f8a9c102370 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa43f4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c10cf80 con 0x7f8a9c0fa9c0 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa198f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c102370 0x7f8a9c113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa198f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c102370 0x7f8a9c113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60018/0 (socket says 172.21.15.86:60018) 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa2190640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c0fa9c0 0x7f8a9c113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:06.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.943+0000 7f8aa198f640 1 -- 172.21.15.86:0/1804598706 learned_addr learned my addr 172.21.15.86:0/1804598706 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.944+0000 7f8aa198f640 1 -- 172.21.15.86:0/1804598706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c0fa9c0 msgr2=0x7f8a9c113000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.944+0000 7f8aa198f640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c0fa9c0 0x7f8a9c113000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.944+0000 7f8aa198f640 1 -- 172.21.15.86:0/1804598706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a98005040 con 0x7f8a9c102370 2024-06-06T08:31:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.944+0000 7f8aa198f640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c102370 0x7f8a9c113540 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a98032430 tx=0x7f8a98009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.944+0000 7f8a92ffd640 1 -- 172.21.15.86:0/1804598706 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a98038070 con 0x7f8a9c102370 2024-06-06T08:31:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.944+0000 7f8a92ffd640 1 -- 172.21.15.86:0/1804598706 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a98008ee0 con 0x7f8a9c102370 2024-06-06T08:31:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.944+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c10d200 con 0x7f8a9c102370 2024-06-06T08:31:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.944+0000 7f8a92ffd640 1 -- 172.21.15.86:0/1804598706 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a9803d7a0 con 0x7f8a9c102370 2024-06-06T08:31:06.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.944+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c10d740 con 0x7f8a9c102370 2024-06-06T08:31:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.945+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c065570 con 0x7f8a9c102370 2024-06-06T08:31:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.945+0000 7f8a92ffd640 1 -- 172.21.15.86:0/1804598706 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f8a98030020 con 0x7f8a9c102370 2024-06-06T08:31:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.946+0000 7f8a92ffd640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8a88065430 0x7f8a880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:06.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.946+0000 7f8a92ffd640 1 -- 172.21.15.86:0/1804598706 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8a980a29b0 con 0x7f8a9c102370 2024-06-06T08:31:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.946+0000 7f8aa2190640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8a88065430 0x7f8a880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.946+0000 7f8aa2190640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8a88065430 0x7f8a880678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c007fd0 tx=0x7f8a8c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:06.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:06.950+0000 7f8a92ffd640 1 -- 172.21.15.86:0/1804598706 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8a98070fa0 con 0x7f8a9c102370 2024-06-06T08:31:07.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.153+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8a9c1094b0 con 0x7f8a9c102370 2024-06-06T08:31:07.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.154+0000 7f8a92ffd640 1 -- 172.21.15.86:0/1804598706 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8a980745c0 con 0x7f8a9c102370 2024-06-06T08:31:07.153 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:07.153 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8a88065430 msgr2=0x7f8a880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8a88065430 0x7f8a880678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c007fd0 tx=0x7f8a8c009040 comp rx=0 tx=0).stop 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c102370 msgr2=0x7f8a9c113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c102370 0x7f8a9c113540 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a98032430 tx=0x7f8a98009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 shutdown_connections 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8a88065430 0x7f8a880678e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a9c102370 0x7f8a9c113540 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 --2- 172.21.15.86:0/1804598706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c0fa9c0 0x7f8a9c113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 >> 172.21.15.86:0/1804598706 conn(0x7f8a9c0709e0 msgr2=0x7f8a9c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 shutdown_connections 2024-06-06T08:31:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:07.156+0000 7f8aa43f4640 1 -- 172.21.15.86:0/1804598706 wait complete. 2024-06-06T08:31:08.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:07 smithi086 ceph-mon[29114]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:08.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:07 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1804598706' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:08.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:07 smithi154 ceph-mon[35428]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:08.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:07 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1804598706' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:10.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:09 smithi086 ceph-mon[29114]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:10.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:09 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:31:10.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:09 smithi154 ceph-mon[35428]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:10.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:09 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:31:10.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:12.211 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:11 smithi086 ceph-mon[29114]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:12.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.387+0000 7f122af5a640 1 -- 172.21.15.86:0/1184050612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fd190 msgr2=0x7f12240fd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:12.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 --2- 172.21.15.86:0/1184050612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fd190 0x7f12240fd560 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f12200048d0 tx=0x7f1220009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:12.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 -- 172.21.15.86:0/1184050612 shutdown_connections 2024-06-06T08:31:12.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 --2- 172.21.15.86:0/1184050612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240fba50 0x7f12240fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:12.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 --2- 172.21.15.86:0/1184050612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fd190 0x7f12240fd560 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 -- 172.21.15.86:0/1184050612 >> 172.21.15.86:0/1184050612 conn(0x7f1224071860 msgr2=0x7f1224071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 -- 172.21.15.86:0/1184050612 shutdown_connections 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 -- 172.21.15.86:0/1184050612 wait complete. 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 Processor -- start 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 -- start start 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.388+0000 7f122af5a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fba50 0x7f12240f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f122af5a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240fd190 0x7f12240f7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f122af5a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12240f84c0 con 0x7f12240fba50 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f122af5a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12240f8630 con 0x7f12240fd190 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f1228cf6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fba50 0x7f12240f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f1228cf6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fba50 0x7f12240f79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52308/0 (socket says 172.21.15.86:52308) 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f1228cf6640 1 -- 172.21.15.86:0/4002539590 learned_addr learned my addr 172.21.15.86:0/4002539590 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f121bfff640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240fd190 0x7f12240f7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:12.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f1228cf6640 1 -- 172.21.15.86:0/4002539590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240fd190 msgr2=0x7f12240f7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:12.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f1228cf6640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240fd190 0x7f12240f7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:12.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f1228cf6640 1 -- 172.21.15.86:0/4002539590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1220005040 con 0x7f12240fba50 2024-06-06T08:31:12.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f1228cf6640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fba50 0x7f12240f79f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1220003080 tx=0x7f1220009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:12.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f12197fa640 1 -- 172.21.15.86:0/4002539590 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1220038070 con 0x7f12240fba50 2024-06-06T08:31:12.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f12197fa640 1 -- 172.21.15.86:0/4002539590 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1220008ee0 con 0x7f12240fba50 2024-06-06T08:31:12.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1224119350 con 0x7f12240fba50 2024-06-06T08:31:12.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f12197fa640 1 -- 172.21.15.86:0/4002539590 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f122003d870 con 0x7f12240fba50 2024-06-06T08:31:12.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.389+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12241196d0 con 0x7f12240fba50 2024-06-06T08:31:12.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.390+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12240fca60 con 0x7f12240fba50 2024-06-06T08:31:12.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.391+0000 7f12197fa640 1 -- 172.21.15.86:0/4002539590 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f1220030020 con 0x7f12240fba50 2024-06-06T08:31:12.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.391+0000 7f12197fa640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f120c065430 0x7f120c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:12.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.391+0000 7f12197fa640 1 -- 172.21.15.86:0/4002539590 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f12200a2190 con 0x7f12240fba50 2024-06-06T08:31:12.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.391+0000 7f121bfff640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f120c065430 0x7f120c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:12.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.391+0000 7f121bfff640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f120c065430 0x7f120c0678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12240f9330 tx=0x7f1214009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:12.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.395+0000 7f12197fa640 1 -- 172.21.15.86:0/4002539590 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1220070760 con 0x7f12240fba50 2024-06-06T08:31:12.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:11 smithi154 ceph-mon[35428]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:12.593 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:12.593 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:12.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.594+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f12240f87f0 con 0x7f12240fba50 2024-06-06T08:31:12.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.594+0000 7f12197fa640 1 -- 172.21.15.86:0/4002539590 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1220073d80 con 0x7f12240fba50 2024-06-06T08:31:12.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f120c065430 msgr2=0x7f120c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:12.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f120c065430 0x7f120c0678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12240f9330 tx=0x7f1214009040 comp rx=0 tx=0).stop 2024-06-06T08:31:12.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fba50 msgr2=0x7f12240f79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:12.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fba50 0x7f12240f79f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1220003080 tx=0x7f1220009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:12.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 shutdown_connections 2024-06-06T08:31:12.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240fd190 0x7f12240f7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:12.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f120c065430 0x7f120c0678e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:12.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 --2- 172.21.15.86:0/4002539590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12240fba50 0x7f12240f79f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:12.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 >> 172.21.15.86:0/4002539590 conn(0x7f1224071860 msgr2=0x7f12240fa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:12.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 shutdown_connections 2024-06-06T08:31:12.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:12.596+0000 7f122af5a640 1 -- 172.21.15.86:0/4002539590 wait complete. 2024-06-06T08:31:13.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:12 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/4002539590' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:13.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:12 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/4002539590' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:14.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:13 smithi086 ceph-mon[29114]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:14.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:13 smithi154 ceph-mon[35428]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:16.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:16.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:15 smithi086 ceph-mon[29114]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:16.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:15 smithi154 ceph-mon[35428]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:16.423 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:17.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.893+0000 7fe398a2f640 1 -- 172.21.15.86:0/1753627972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3941022d0 msgr2=0x7fe3941026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:17.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.893+0000 7fe398a2f640 1 --2- 172.21.15.86:0/1753627972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3941022d0 0x7fe3941026a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe38c0048d0 tx=0x7fe38c009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:17.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.893+0000 7fe398a2f640 1 -- 172.21.15.86:0/1753627972 shutdown_connections 2024-06-06T08:31:17.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.893+0000 7fe398a2f640 1 --2- 172.21.15.86:0/1753627972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe394102c70 0x7fe3940fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:17.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.893+0000 7fe398a2f640 1 --2- 172.21.15.86:0/1753627972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3941022d0 0x7fe3941026a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:17.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.893+0000 7fe398a2f640 1 -- 172.21.15.86:0/1753627972 >> 172.21.15.86:0/1753627972 conn(0x7fe3940f7e80 msgr2=0x7fe3940fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe398a2f640 1 -- 172.21.15.86:0/1753627972 shutdown_connections 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe398a2f640 1 -- 172.21.15.86:0/1753627972 wait complete. 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe398a2f640 1 Processor -- start 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe398a2f640 1 -- start start 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe398a2f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3941022d0 0x7fe394115290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe398a2f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe394102c70 0x7fe3941157d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe398a2f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394115e60 con 0x7fe394102c70 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe398a2f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39410f210 con 0x7fe3941022d0 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe391d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe394102c70 0x7fe3941157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe391d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe394102c70 0x7fe3941157d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37232/0 (socket says 172.21.15.86:37232) 2024-06-06T08:31:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.894+0000 7fe391d9b640 1 -- 172.21.15.86:0/776904138 learned_addr learned my addr 172.21.15.86:0/776904138 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe39259c640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3941022d0 0x7fe394115290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe391d9b640 1 -- 172.21.15.86:0/776904138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3941022d0 msgr2=0x7fe394115290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe391d9b640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3941022d0 0x7fe394115290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe391d9b640 1 -- 172.21.15.86:0/776904138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c005040 con 0x7fe394102c70 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe391d9b640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe394102c70 0x7fe3941157d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe38800b790 tx=0x7fe3880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe382ffd640 1 -- 172.21.15.86:0/776904138 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe38800c700 con 0x7fe394102c70 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe382ffd640 1 -- 172.21.15.86:0/776904138 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe38800cd30 con 0x7fe394102c70 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39410f4f0 con 0x7fe394102c70 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe382ffd640 1 -- 172.21.15.86:0/776904138 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe388013370 con 0x7fe394102c70 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.895+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39410fa10 con 0x7fe394102c70 2024-06-06T08:31:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.896+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3940fda40 con 0x7fe394102c70 2024-06-06T08:31:17.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.896+0000 7fe382ffd640 1 -- 172.21.15.86:0/776904138 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe388012070 con 0x7fe394102c70 2024-06-06T08:31:17.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.897+0000 7fe382ffd640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe37c065430 0x7fe37c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:17.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.897+0000 7fe382ffd640 1 -- 172.21.15.86:0/776904138 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe388081f00 con 0x7fe394102c70 2024-06-06T08:31:17.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.897+0000 7fe39259c640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe37c065430 0x7fe37c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:17.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.897+0000 7fe39259c640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe37c065430 0x7fe37c0678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe38c000c00 tx=0x7fe38c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:17.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:17.901+0000 7fe382ffd640 1 -- 172.21.15.86:0/776904138 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe3880503b0 con 0x7fe394102c70 2024-06-06T08:31:18.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.099+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe3941094c0 con 0x7fe394102c70 2024-06-06T08:31:18.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.099+0000 7fe382ffd640 1 -- 172.21.15.86:0/776904138 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe3880539d0 con 0x7fe394102c70 2024-06-06T08:31:18.099 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:18.099 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:18.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.101+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe37c065430 msgr2=0x7fe37c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.101+0000 7fe398a2f640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe37c065430 0x7fe37c0678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe38c000c00 tx=0x7fe38c035040 comp rx=0 tx=0).stop 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.101+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe394102c70 msgr2=0x7fe3941157d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.101+0000 7fe398a2f640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe394102c70 0x7fe3941157d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe38800b790 tx=0x7fe3880091d0 comp rx=0 tx=0).stop 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.101+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 shutdown_connections 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.101+0000 7fe398a2f640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fe37c065430 0x7fe37c0678e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.101+0000 7fe398a2f640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe394102c70 0x7fe3941157d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.101+0000 7fe398a2f640 1 --2- 172.21.15.86:0/776904138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3941022d0 0x7fe394115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.101+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 >> 172.21.15.86:0/776904138 conn(0x7fe3940f7e80 msgr2=0x7fe394071170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.102+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 shutdown_connections 2024-06-06T08:31:18.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:18.102+0000 7fe398a2f640 1 -- 172.21.15.86:0/776904138 wait complete. 2024-06-06T08:31:18.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:17 smithi086 ceph-mon[29114]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:18.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:17 smithi154 ceph-mon[35428]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:19.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:18 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/776904138' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:19.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:18 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/776904138' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:20.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:19 smithi086 ceph-mon[29114]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:20.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:19 smithi154 ceph-mon[35428]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:21.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:21.887 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:22.238 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:21 smithi086 ceph-mon[29114]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:22.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:21 smithi154 ceph-mon[35428]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:23.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.171+0000 7f4f4b798640 1 -- 172.21.15.86:0/672084531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f441011d0 msgr2=0x7f4f441015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.171+0000 7f4f4b798640 1 --2- 172.21.15.86:0/672084531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f441011d0 0x7f4f441015a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0048d0 tx=0x7f4f3c009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.171+0000 7f4f4b798640 1 -- 172.21.15.86:0/672084531 shutdown_connections 2024-06-06T08:31:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.171+0000 7f4f4b798640 1 --2- 172.21.15.86:0/672084531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f44101b70 0x7f4f440fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.171+0000 7f4f4b798640 1 --2- 172.21.15.86:0/672084531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f441011d0 0x7f4f441015a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.171+0000 7f4f4b798640 1 -- 172.21.15.86:0/672084531 >> 172.21.15.86:0/672084531 conn(0x7f4f440f6dc0 msgr2=0x7f4f440f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f4b798640 1 -- 172.21.15.86:0/672084531 shutdown_connections 2024-06-06T08:31:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f4b798640 1 -- 172.21.15.86:0/672084531 wait complete. 2024-06-06T08:31:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f4b798640 1 Processor -- start 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f4b798640 1 -- start start 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f4b798640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f441011d0 0x7f4f4410e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f4b798640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f44101b70 0x7f4f4410e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f4b798640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4410ec70 con 0x7f4f44101b70 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f4b798640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4410ede0 con 0x7f4f441011d0 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f48d33640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f44101b70 0x7f4f4410e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.172+0000 7f4f48d33640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f44101b70 0x7f4f4410e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37244/0 (socket says 172.21.15.86:37244) 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f48d33640 1 -- 172.21.15.86:0/3888075388 learned_addr learned my addr 172.21.15.86:0/3888075388 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f49534640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f441011d0 0x7f4f4410e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f48d33640 1 -- 172.21.15.86:0/3888075388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f441011d0 msgr2=0x7f4f4410e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f48d33640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f441011d0 0x7f4f4410e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:23.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f48d33640 1 -- 172.21.15.86:0/3888075388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c005040 con 0x7f4f44101b70 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f48d33640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f44101b70 0x7f4f4410e590 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800b790 tx=0x7f4f380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f41ffb640 1 -- 172.21.15.86:0/3888075388 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f3800c700 con 0x7f4f44101b70 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f41ffb640 1 -- 172.21.15.86:0/3888075388 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f3800cd30 con 0x7f4f44101b70 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f41ffb640 1 -- 172.21.15.86:0/3888075388 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f38013370 con 0x7f4f44101b70 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f441170c0 con 0x7f4f44101b70 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.173+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f441175e0 con 0x7f4f44101b70 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.174+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f440fc940 con 0x7f4f44101b70 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.175+0000 7f4f41ffb640 1 -- 172.21.15.86:0/3888075388 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4f38012070 con 0x7f4f44101b70 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.175+0000 7f4f41ffb640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4f2c065430 0x7f4f2c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.175+0000 7f4f41ffb640 1 -- 172.21.15.86:0/3888075388 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4f38081cd0 con 0x7f4f44101b70 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.175+0000 7f4f49534640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4f2c065430 0x7f4f2c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.175+0000 7f4f49534640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4f2c065430 0x7f4f2c0678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009f80 tx=0x7f4f3c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:23.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.179+0000 7f4f41ffb640 1 -- 172.21.15.86:0/3888075388 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4f38050200 con 0x7f4f44101b70 2024-06-06T08:31:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.385+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4f4404b480 con 0x7f4f44101b70 2024-06-06T08:31:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.386+0000 7f4f41ffb640 1 -- 172.21.15.86:0/3888075388 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4f38053820 con 0x7f4f44101b70 2024-06-06T08:31:23.385 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:23.385 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4f2c065430 msgr2=0x7f4f2c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4f2c065430 0x7f4f2c0678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009f80 tx=0x7f4f3c035040 comp rx=0 tx=0).stop 2024-06-06T08:31:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f44101b70 msgr2=0x7f4f4410e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f44101b70 0x7f4f4410e590 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800b790 tx=0x7f4f380091d0 comp rx=0 tx=0).stop 2024-06-06T08:31:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 shutdown_connections 2024-06-06T08:31:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f4f2c065430 0x7f4f2c0678e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f44101b70 0x7f4f4410e590 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 --2- 172.21.15.86:0/3888075388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f441011d0 0x7f4f4410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:23.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 >> 172.21.15.86:0/3888075388 conn(0x7f4f440f6dc0 msgr2=0x7f4f440f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:23.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 shutdown_connections 2024-06-06T08:31:23.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:23.388+0000 7f4f4b798640 1 -- 172.21.15.86:0/3888075388 wait complete. 2024-06-06T08:31:24.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:23 smithi086 ceph-mon[29114]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:24.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:23 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/3888075388' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:24.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:23 smithi154 ceph-mon[35428]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:24.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:23 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/3888075388' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:25.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:24 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:31:25.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:24 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:31:26.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:25 smithi086 ceph-mon[29114]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:26.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:25 smithi154 ceph-mon[35428]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:27.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:27.273 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:28.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:27 smithi086 ceph-mon[29114]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:28.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:27 smithi154 ceph-mon[35428]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.802+0000 7f0582a06640 1 -- 172.21.15.86:0/3113357791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 msgr2=0x7f057c0fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.802+0000 7f0582a06640 1 --2- 172.21.15.86:0/3113357791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 0x7f057c0fb200 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f05640048d0 tx=0x7f0564009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.802+0000 7f0582a06640 1 -- 172.21.15.86:0/3113357791 shutdown_connections 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.802+0000 7f0582a06640 1 --2- 172.21.15.86:0/3113357791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 0x7f057c0fb200 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.802+0000 7f0582a06640 1 --2- 172.21.15.86:0/3113357791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c1011d0 0x7f057c1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.802+0000 7f0582a06640 1 -- 172.21.15.86:0/3113357791 >> 172.21.15.86:0/3113357791 conn(0x7f057c0f6d80 msgr2=0x7f057c0f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.802+0000 7f0582a06640 1 -- 172.21.15.86:0/3113357791 shutdown_connections 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.802+0000 7f0582a06640 1 -- 172.21.15.86:0/3113357791 wait complete. 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.803+0000 7f0582a06640 1 Processor -- start 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.803+0000 7f0582a06640 1 -- start start 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0582a06640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c1011d0 0x7f057c111f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0582a06640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 0x7f057c1124c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0582a06640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c112b50 con 0x7f057c101b70 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0582a06640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c10bf00 con 0x7f057c1011d0 2024-06-06T08:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0573fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 0x7f057c1124c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0573fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 0x7f057c1124c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46122/0 (socket says 172.21.15.86:46122) 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0573fff640 1 -- 172.21.15.86:0/951103516 learned_addr learned my addr 172.21.15.86:0/951103516 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f057bfff640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c1011d0 0x7f057c111f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0573fff640 1 -- 172.21.15.86:0/951103516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c1011d0 msgr2=0x7f057c111f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0573fff640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c1011d0 0x7f057c111f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0573fff640 1 -- 172.21.15.86:0/951103516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0564005040 con 0x7f057c101b70 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0573fff640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 0x7f057c1124c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0564003080 tx=0x7f0564009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f05797fa640 1 -- 172.21.15.86:0/951103516 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056403a070 con 0x7f057c101b70 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f05797fa640 1 -- 172.21.15.86:0/951103516 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0564008ee0 con 0x7f057c101b70 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f05797fa640 1 -- 172.21.15.86:0/951103516 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0564032a40 con 0x7f057c101b70 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c10c180 con 0x7f057c101b70 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.804+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c10c640 con 0x7f057c101b70 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.805+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c0fc940 con 0x7f057c101b70 2024-06-06T08:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.806+0000 7f05797fa640 1 -- 172.21.15.86:0/951103516 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0564030020 con 0x7f057c101b70 2024-06-06T08:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.806+0000 7f05797fa640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0568065430 0x7f05680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.806+0000 7f05797fa640 1 -- 172.21.15.86:0/951103516 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f05640a4440 con 0x7f057c101b70 2024-06-06T08:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.806+0000 7f057bfff640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0568065430 0x7f05680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.807+0000 7f057bfff640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0568065430 0x7f05680678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f056c009950 tx=0x7f056c008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:28.810+0000 7f05797fa640 1 -- 172.21.15.86:0/951103516 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0564072970 con 0x7f057c101b70 2024-06-06T08:31:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.007+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f057c1083c0 con 0x7f057c101b70 2024-06-06T08:31:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.007+0000 7f05797fa640 1 -- 172.21.15.86:0/951103516 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0564035d90 con 0x7f057c101b70 2024-06-06T08:31:29.007 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:29.007 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0568065430 msgr2=0x7f05680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0568065430 0x7f05680678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f056c009950 tx=0x7f056c008040 comp rx=0 tx=0).stop 2024-06-06T08:31:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 msgr2=0x7f057c1124c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 0x7f057c1124c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0564003080 tx=0x7f0564009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 shutdown_connections 2024-06-06T08:31:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f0568065430 0x7f05680678e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f057c101b70 0x7f057c1124c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 --2- 172.21.15.86:0/951103516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c1011d0 0x7f057c111f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 >> 172.21.15.86:0/951103516 conn(0x7f057c0f6d80 msgr2=0x7f057c0f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.009+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 shutdown_connections 2024-06-06T08:31:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:29.010+0000 7f0582a06640 1 -- 172.21.15.86:0/951103516 wait complete. 2024-06-06T08:31:30.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:30 smithi086 ceph-mon[29114]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:30.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:30 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/951103516' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:30 smithi154 ceph-mon[35428]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:30 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/951103516' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:32.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:32 smithi086 ceph-mon[29114]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:32.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:32 smithi154 ceph-mon[35428]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:32.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:32.818 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:34.077 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:34 smithi086 ceph-mon[29114]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:34.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.237+0000 7f3505330640 1 -- 172.21.15.86:0/280927451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 msgr2=0x7f35000fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:34.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.237+0000 7f3505330640 1 --2- 172.21.15.86:0/280927451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 0x7f35000fc2f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f34f40048d0 tx=0x7f34f4009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:34.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.237+0000 7f3505330640 1 -- 172.21.15.86:0/280927451 shutdown_connections 2024-06-06T08:31:34.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.237+0000 7f3505330640 1 --2- 172.21.15.86:0/280927451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 0x7f35000fc2f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:34.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.237+0000 7f3505330640 1 --2- 172.21.15.86:0/280927451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35001022c0 0x7f3500102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:34.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.237+0000 7f3505330640 1 -- 172.21.15.86:0/280927451 >> 172.21.15.86:0/280927451 conn(0x7f35000f7e50 msgr2=0x7f35000fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:34.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.237+0000 7f3505330640 1 -- 172.21.15.86:0/280927451 shutdown_connections 2024-06-06T08:31:34.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.237+0000 7f3505330640 1 -- 172.21.15.86:0/280927451 wait complete. 2024-06-06T08:31:34.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f3505330640 1 Processor -- start 2024-06-06T08:31:34.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f3505330640 1 -- start start 2024-06-06T08:31:34.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f3505330640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35001022c0 0x7f3500113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:34.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f3505330640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 0x7f3500113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:34.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f3505330640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3500113e90 con 0x7f3500102c60 2024-06-06T08:31:34.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f3505330640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350010d2d0 con 0x7f35001022c0 2024-06-06T08:31:34.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f34fe7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 0x7f3500113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:34.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f34fe7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 0x7f3500113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46148/0 (socket says 172.21.15.86:46148) 2024-06-06T08:31:34.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f34fe7fc640 1 -- 172.21.15.86:0/1176412814 learned_addr learned my addr 172.21.15.86:0/1176412814 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:34.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f34feffd640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35001022c0 0x7f3500113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:34.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f34fe7fc640 1 -- 172.21.15.86:0/1176412814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35001022c0 msgr2=0x7f3500113350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:34.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f34fe7fc640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35001022c0 0x7f3500113350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:34.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f34fe7fc640 1 -- 172.21.15.86:0/1176412814 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f4005040 con 0x7f3500102c60 2024-06-06T08:31:34.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.238+0000 7f34fe7fc640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 0x7f3500113890 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f34f40043d0 tx=0x7f34f4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:34.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.239+0000 7f34e77fe640 1 -- 172.21.15.86:0/1176412814 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f4038070 con 0x7f3500102c60 2024-06-06T08:31:34.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.239+0000 7f34e77fe640 1 -- 172.21.15.86:0/1176412814 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34f4032970 con 0x7f3500102c60 2024-06-06T08:31:34.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.239+0000 7f34e77fe640 1 -- 172.21.15.86:0/1176412814 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f403da60 con 0x7f3500102c60 2024-06-06T08:31:34.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.239+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350010d550 con 0x7f3500102c60 2024-06-06T08:31:34.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.239+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350010da90 con 0x7f3500102c60 2024-06-06T08:31:34.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.240+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35000fda30 con 0x7f3500102c60 2024-06-06T08:31:34.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.240+0000 7f34e77fe640 1 -- 172.21.15.86:0/1176412814 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f34f4030020 con 0x7f3500102c60 2024-06-06T08:31:34.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.241+0000 7f34e77fe640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f34e8065430 0x7f34e80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:34.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.241+0000 7f34e77fe640 1 -- 172.21.15.86:0/1176412814 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f34f40a2500 con 0x7f3500102c60 2024-06-06T08:31:34.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.241+0000 7f34feffd640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f34e8065430 0x7f34e80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:34.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.241+0000 7f34feffd640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f34e8065430 0x7f34e80678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f34f0004ed0 tx=0x7f34f000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:34.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.245+0000 7f34e77fe640 1 -- 172.21.15.86:0/1176412814 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f34f40709b0 con 0x7f3500102c60 2024-06-06T08:31:34.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:34 smithi154 ceph-mon[35428]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:34.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.443+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3500109420 con 0x7f3500102c60 2024-06-06T08:31:34.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.444+0000 7f34e77fe640 1 -- 172.21.15.86:0/1176412814 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f34f4073fd0 con 0x7f3500102c60 2024-06-06T08:31:34.443 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:34.443 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:34.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f34e8065430 msgr2=0x7f34e80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:34.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f34e8065430 0x7f34e80678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f34f0004ed0 tx=0x7f34f000b040 comp rx=0 tx=0).stop 2024-06-06T08:31:34.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 msgr2=0x7f3500113890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:34.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 0x7f3500113890 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f34f40043d0 tx=0x7f34f4009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:34.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 shutdown_connections 2024-06-06T08:31:34.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f34e8065430 0x7f34e80678e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:34.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3500102c60 0x7f3500113890 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:34.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 --2- 172.21.15.86:0/1176412814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35001022c0 0x7f3500113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:34.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 >> 172.21.15.86:0/1176412814 conn(0x7f35000f7e50 msgr2=0x7f35000f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:34.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 shutdown_connections 2024-06-06T08:31:34.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:34.446+0000 7f3505330640 1 -- 172.21.15.86:0/1176412814 wait complete. 2024-06-06T08:31:35.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:35 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1176412814' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:35.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:35 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1176412814' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:36.318 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:36 smithi154 ceph-mon[35428]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:36.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:36 smithi086 ceph-mon[29114]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:37.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:38.244 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:38.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:38 smithi086 ceph-mon[29114]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:38.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:38 smithi154 ceph-mon[35428]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:39.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.524+0000 7f7e8b093640 1 -- 172.21.15.86:0/4202464284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 msgr2=0x7f7e84065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:39.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.524+0000 7f7e8b093640 1 --2- 172.21.15.86:0/4202464284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 0x7f7e84065470 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7e740048d0 tx=0x7f7e74009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:39.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.524+0000 7f7e8b093640 1 -- 172.21.15.86:0/4202464284 shutdown_connections 2024-06-06T08:31:39.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.524+0000 7f7e8b093640 1 --2- 172.21.15.86:0/4202464284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e840fdeb0 0x7f7e840fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:39.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.524+0000 7f7e8b093640 1 --2- 172.21.15.86:0/4202464284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 0x7f7e84065470 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:39.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.524+0000 7f7e8b093640 1 -- 172.21.15.86:0/4202464284 >> 172.21.15.86:0/4202464284 conn(0x7f7e84072110 msgr2=0x7f7e84072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:39.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.524+0000 7f7e8b093640 1 -- 172.21.15.86:0/4202464284 shutdown_connections 2024-06-06T08:31:39.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.524+0000 7f7e8b093640 1 -- 172.21.15.86:0/4202464284 wait complete. 2024-06-06T08:31:39.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.525+0000 7f7e8b093640 1 Processor -- start 2024-06-06T08:31:39.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.525+0000 7f7e8b093640 1 -- start start 2024-06-06T08:31:39.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.525+0000 7f7e8b093640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 0x7f7e841152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:39.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.525+0000 7f7e8b093640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e840fdeb0 0x7f7e841157e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:39.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.525+0000 7f7e8b093640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e84115e70 con 0x7f7e840650a0 2024-06-06T08:31:39.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.525+0000 7f7e8b093640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8410f220 con 0x7f7e840fdeb0 2024-06-06T08:31:39.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.525+0000 7f7e88e2f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 0x7f7e841152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:39.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.525+0000 7f7e88e2f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 0x7f7e841152a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33730/0 (socket says 172.21.15.86:33730) 2024-06-06T08:31:39.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.525+0000 7f7e88e2f640 1 -- 172.21.15.86:0/2652688773 learned_addr learned my addr 172.21.15.86:0/2652688773 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e88e2f640 1 -- 172.21.15.86:0/2652688773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e840fdeb0 msgr2=0x7f7e841157e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e83fff640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e840fdeb0 0x7f7e841157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e88e2f640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e840fdeb0 0x7f7e841157e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e88e2f640 1 -- 172.21.15.86:0/2652688773 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74005040 con 0x7f7e840650a0 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e83fff640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e840fdeb0 0x7f7e841157e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e88e2f640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 0x7f7e841152a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7e74003080 tx=0x7f7e74009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e817fa640 1 -- 172.21.15.86:0/2652688773 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e74038070 con 0x7f7e840650a0 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8410f4a0 con 0x7f7e840650a0 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e817fa640 1 -- 172.21.15.86:0/2652688773 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e74008ee0 con 0x7f7e840650a0 2024-06-06T08:31:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e817fa640 1 -- 172.21.15.86:0/2652688773 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e7403d870 con 0x7f7e840650a0 2024-06-06T08:31:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.526+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8410f960 con 0x7f7e840650a0 2024-06-06T08:31:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.527+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e840fda50 con 0x7f7e840650a0 2024-06-06T08:31:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.527+0000 7f7e817fa640 1 -- 172.21.15.86:0/2652688773 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f7e74030020 con 0x7f7e840650a0 2024-06-06T08:31:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.528+0000 7f7e817fa640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e6c065430 0x7f7e6c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.528+0000 7f7e817fa640 1 -- 172.21.15.86:0/2652688773 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7e740a2240 con 0x7f7e840650a0 2024-06-06T08:31:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.528+0000 7f7e83fff640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e6c065430 0x7f7e6c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.528+0000 7f7e83fff640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e6c065430 0x7f7e6c0678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7e84110b60 tx=0x7f7e78008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:39.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.532+0000 7f7e817fa640 1 -- 172.21.15.86:0/2652688773 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7e74070770 con 0x7f7e840650a0 2024-06-06T08:31:39.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.731+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7e8404b4d0 con 0x7f7e840650a0 2024-06-06T08:31:39.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.732+0000 7f7e817fa640 1 -- 172.21.15.86:0/2652688773 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7e74073d90 con 0x7f7e840650a0 2024-06-06T08:31:39.731 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:39.731 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:39.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e6c065430 msgr2=0x7f7e6c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:39.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e6c065430 0x7f7e6c0678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7e84110b60 tx=0x7f7e78008040 comp rx=0 tx=0).stop 2024-06-06T08:31:39.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 msgr2=0x7f7e841152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:39.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 0x7f7e841152a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7e74003080 tx=0x7f7e74009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:39.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 shutdown_connections 2024-06-06T08:31:39.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e840fdeb0 0x7f7e841157e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:39.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7e6c065430 0x7f7e6c0678e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:39.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 --2- 172.21.15.86:0/2652688773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e840650a0 0x7f7e841152a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:39.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 >> 172.21.15.86:0/2652688773 conn(0x7f7e84072110 msgr2=0x7f7e841010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:39.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 shutdown_connections 2024-06-06T08:31:39.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:39.734+0000 7f7e8b093640 1 -- 172.21.15.86:0/2652688773 wait complete. 2024-06-06T08:31:40.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:40 smithi086 ceph-mon[29114]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:40.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:31:40.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:40 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2652688773' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:40.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:40 smithi154 ceph-mon[35428]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:40.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:40 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:31:40.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:40 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2652688773' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:42.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:42 smithi086 ceph-mon[29114]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:42.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:42 smithi154 ceph-mon[35428]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:43.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:43.575 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:44.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:44 smithi086 ceph-mon[29114]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:44.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:44 smithi154 ceph-mon[35428]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:45.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 -- 172.21.15.86:0/684958794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 msgr2=0x7ff8d80faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:45.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 --2- 172.21.15.86:0/684958794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 0x7ff8d80faec0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40048d0 tx=0x7ff8d4009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:45.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 -- 172.21.15.86:0/684958794 shutdown_connections 2024-06-06T08:31:45.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 --2- 172.21.15.86:0/684958794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 0x7ff8d80faec0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:45.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 --2- 172.21.15.86:0/684958794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d81023e0 0x7ff8d81027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:45.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 -- 172.21.15.86:0/684958794 >> 172.21.15.86:0/684958794 conn(0x7ff8d8070ab0 msgr2=0x7ff8d8070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:45.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 -- 172.21.15.86:0/684958794 shutdown_connections 2024-06-06T08:31:45.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 -- 172.21.15.86:0/684958794 wait complete. 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 Processor -- start 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.103+0000 7ff8e02b0640 1 -- start start 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8e02b0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 0x7ff8d81130e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8e02b0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d81023e0 0x7ff8d8113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8e02b0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d8113c20 con 0x7ff8d80faa50 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8e02b0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d810d060 con 0x7ff8d81023e0 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8de04c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 0x7ff8d81130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8de04c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 0x7ff8d81130e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33752/0 (socket says 172.21.15.86:33752) 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8de04c640 1 -- 172.21.15.86:0/1496923795 learned_addr learned my addr 172.21.15.86:0/1496923795 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8dd84b640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d81023e0 0x7ff8d8113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8de04c640 1 -- 172.21.15.86:0/1496923795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d81023e0 msgr2=0x7ff8d8113620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8de04c640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d81023e0 0x7ff8d8113620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8de04c640 1 -- 172.21.15.86:0/1496923795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d4005040 con 0x7ff8d80faa50 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8de04c640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 0x7ff8d81130e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8002a50 tx=0x7ff8c8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8ceffd640 1 -- 172.21.15.86:0/1496923795 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c8007c30 con 0x7ff8d80faa50 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8ceffd640 1 -- 172.21.15.86:0/1496923795 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8c800f040 con 0x7ff8d80faa50 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d810d340 con 0x7ff8d80faa50 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8ceffd640 1 -- 172.21.15.86:0/1496923795 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c800dad0 con 0x7ff8d80faa50 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.104+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d810d830 con 0x7ff8d80faa50 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.105+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d8065550 con 0x7ff8d80faa50 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.106+0000 7ff8ceffd640 1 -- 172.21.15.86:0/1496923795 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff8c800dd40 con 0x7ff8d80faa50 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.106+0000 7ff8ceffd640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff8c4065430 0x7ff8c40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.106+0000 7ff8ceffd640 1 -- 172.21.15.86:0/1496923795 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff8c8081c90 con 0x7ff8d80faa50 2024-06-06T08:31:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.106+0000 7ff8dd84b640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff8c4065430 0x7ff8c40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:45.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.106+0000 7ff8dd84b640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff8c4065430 0x7ff8c40678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009f80 tx=0x7ff8d4035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:45.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.110+0000 7ff8ceffd640 1 -- 172.21.15.86:0/1496923795 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff8c804cdf0 con 0x7ff8d80faa50 2024-06-06T08:31:45.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.306+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff8d804b480 con 0x7ff8d80faa50 2024-06-06T08:31:45.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.307+0000 7ff8ceffd640 1 -- 172.21.15.86:0/1496923795 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff8c80501f0 con 0x7ff8d80faa50 2024-06-06T08:31:45.306 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:45.306 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:45.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff8c4065430 msgr2=0x7ff8c40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:45.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff8c4065430 0x7ff8c40678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009f80 tx=0x7ff8d4035040 comp rx=0 tx=0).stop 2024-06-06T08:31:45.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 msgr2=0x7ff8d81130e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:45.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 0x7ff8d81130e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8002a50 tx=0x7ff8c8004600 comp rx=0 tx=0).stop 2024-06-06T08:31:45.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 shutdown_connections 2024-06-06T08:31:45.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d81023e0 0x7ff8d8113620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:45.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff8c4065430 0x7ff8c40678e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:45.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 --2- 172.21.15.86:0/1496923795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8d80faa50 0x7ff8d81130e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:45.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 >> 172.21.15.86:0/1496923795 conn(0x7ff8d8070ab0 msgr2=0x7ff8d80f9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:45.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 shutdown_connections 2024-06-06T08:31:45.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:45.309+0000 7ff8e02b0640 1 -- 172.21.15.86:0/1496923795 wait complete. 2024-06-06T08:31:46.338 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:46 smithi086 ceph-mon[29114]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:46.338 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:46 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1496923795' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:46.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:46 smithi154 ceph-mon[35428]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:46.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:46 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1496923795' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:48.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:48 smithi086 ceph-mon[29114]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:48.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:48 smithi154 ceph-mon[35428]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:48.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:49.243 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:50.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:50 smithi086 ceph-mon[29114]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:50.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:31:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:31:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:31:50.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:50 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:31:50.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:50 smithi154 ceph-mon[35428]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T08:31:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T08:31:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T08:31:50.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:50 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' 2024-06-06T08:31:50.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.826+0000 7f73575b1640 1 -- 172.21.15.86:0/835549392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 msgr2=0x7f73500fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:50.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.826+0000 7f73575b1640 1 --2- 172.21.15.86:0/835549392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 0x7f73500fc2f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7344009c50 tx=0x7f7344004d10 comp rx=0 tx=0).stop 2024-06-06T08:31:50.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.826+0000 7f73575b1640 1 -- 172.21.15.86:0/835549392 shutdown_connections 2024-06-06T08:31:50.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.826+0000 7f73575b1640 1 --2- 172.21.15.86:0/835549392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 0x7f73500fc2f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:50.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.826+0000 7f73575b1640 1 --2- 172.21.15.86:0/835549392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73501022c0 0x7f7350102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:50.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f73575b1640 1 -- 172.21.15.86:0/835549392 >> 172.21.15.86:0/835549392 conn(0x7f73500f7e50 msgr2=0x7f73500fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f73575b1640 1 -- 172.21.15.86:0/835549392 shutdown_connections 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f73575b1640 1 -- 172.21.15.86:0/835549392 wait complete. 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f73575b1640 1 Processor -- start 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f73575b1640 1 -- start start 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f73575b1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73501022c0 0x7f73501133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f73575b1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 0x7f7350113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f73575b1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7350113f90 con 0x7f7350102c60 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f73575b1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735010d340 con 0x7f73501022c0 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f7354b4c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 0x7f7350113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f7354b4c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 0x7f7350113900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58084/0 (socket says 172.21.15.86:58084) 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.827+0000 7f7354b4c640 1 -- 172.21.15.86:0/2138831219 learned_addr learned my addr 172.21.15.86:0/2138831219 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:50.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f735534d640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73501022c0 0x7f73501133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f7354b4c640 1 -- 172.21.15.86:0/2138831219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73501022c0 msgr2=0x7f73501133c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f7354b4c640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73501022c0 0x7f73501133c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f7354b4c640 1 -- 172.21.15.86:0/2138831219 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73440098b0 con 0x7f7350102c60 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f735534d640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73501022c0 0x7f73501133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f7354b4c640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 0x7f7350113900 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7344004f80 tx=0x7f7344004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f734dffb640 1 -- 172.21.15.86:0/2138831219 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7344008de0 con 0x7f7350102c60 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f734dffb640 1 -- 172.21.15.86:0/2138831219 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7344032c00 con 0x7f7350102c60 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f734dffb640 1 -- 172.21.15.86:0/2138831219 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f734403dbc0 con 0x7f7350102c60 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735010d5c0 con 0x7f7350102c60 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.828+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735010da90 con 0x7f7350102c60 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.829+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73500fda30 con 0x7f7350102c60 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.830+0000 7f734dffb640 1 -- 172.21.15.86:0/2138831219 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f734402f030 con 0x7f7350102c60 2024-06-06T08:31:50.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.830+0000 7f734dffb640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7338065430 0x7f73380678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:50.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.830+0000 7f734dffb640 1 -- 172.21.15.86:0/2138831219 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f73440a3bd0 con 0x7f7350102c60 2024-06-06T08:31:50.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.830+0000 7f735534d640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7338065430 0x7f73380678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:50.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.830+0000 7f735534d640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7338065430 0x7f73380678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7348006fd0 tx=0x7f7348008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:50.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:50.834+0000 7f734dffb640 1 -- 172.21.15.86:0/2138831219 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7344072080 con 0x7f7350102c60 2024-06-06T08:31:51.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.036+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f73501094e0 con 0x7f7350102c60 2024-06-06T08:31:51.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.036+0000 7f734dffb640 1 -- 172.21.15.86:0/2138831219 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7344032d70 con 0x7f7350102c60 2024-06-06T08:31:51.039 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:51.039 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:51.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7338065430 msgr2=0x7f73380678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:51.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7338065430 0x7f73380678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7348006fd0 tx=0x7f7348008040 comp rx=0 tx=0).stop 2024-06-06T08:31:51.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 msgr2=0x7f7350113900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:51.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 0x7f7350113900 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7344004f80 tx=0x7f7344004d10 comp rx=0 tx=0).stop 2024-06-06T08:31:51.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 shutdown_connections 2024-06-06T08:31:51.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f7338065430 0x7f73380678e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:51.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7350102c60 0x7f7350113900 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:51.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 --2- 172.21.15.86:0/2138831219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73501022c0 0x7f73501133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:51.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 >> 172.21.15.86:0/2138831219 conn(0x7f73500f7e50 msgr2=0x7f73500f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:51.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 shutdown_connections 2024-06-06T08:31:51.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:51.038+0000 7f73575b1640 1 -- 172.21.15.86:0/2138831219 wait complete. 2024-06-06T08:31:52.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:52 smithi086 ceph-mon[29114]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:52.335 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:52 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2138831219' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:52.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:52 smithi154 ceph-mon[35428]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:52.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:52 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2138831219' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:54.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:54 smithi086 ceph-mon[29114]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:54.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:54 smithi154 ceph-mon[35428]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:54.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:31:54.874 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:31:55.289 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:55 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:31:55.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:55 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:31:56.147 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:56 smithi086 ceph-mon[29114]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:56.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2824321064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb374101b50 msgr2=0x7fb3740fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:56.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 --2- 172.21.15.86:0/2824321064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb374101b50 0x7fb3740fb1e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb3640048d0 tx=0x7fb364009da0 comp rx=0 tx=0).stop 2024-06-06T08:31:56.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2824321064 shutdown_connections 2024-06-06T08:31:56.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 --2- 172.21.15.86:0/2824321064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb374101b50 0x7fb3740fb1e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:56.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 --2- 172.21.15.86:0/2824321064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3741011b0 0x7fb374101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:56.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2824321064 >> 172.21.15.86:0/2824321064 conn(0x7fb3740f6d60 msgr2=0x7fb3740f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2824321064 shutdown_connections 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2824321064 wait complete. 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 Processor -- start 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 -- start start 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3741011b0 0x7fb374113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb374101b50 0x7fb374113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374113b60 con 0x7fb3741011b0 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb37bcb0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37410cfa0 con 0x7fb374101b50 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.315+0000 7fb379a4c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3741011b0 0x7fb374113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb379a4c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3741011b0 0x7fb374113020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58100/0 (socket says 172.21.15.86:58100) 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb379a4c640 1 -- 172.21.15.86:0/2650597226 learned_addr learned my addr 172.21.15.86:0/2650597226 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb37924b640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb374101b50 0x7fb374113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb379a4c640 1 -- 172.21.15.86:0/2650597226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb374101b50 msgr2=0x7fb374113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:56.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb379a4c640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb374101b50 0x7fb374113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:56.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb379a4c640 1 -- 172.21.15.86:0/2650597226 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb364005040 con 0x7fb3741011b0 2024-06-06T08:31:56.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb379a4c640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3741011b0 0x7fb374113020 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb37000c830 tx=0x7fb370004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:56.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb36a7fc640 1 -- 172.21.15.86:0/2650597226 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb37000a7d0 con 0x7fb3741011b0 2024-06-06T08:31:56.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb36a7fc640 1 -- 172.21.15.86:0/2650597226 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb37000ae00 con 0x7fb3741011b0 2024-06-06T08:31:56.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37410d280 con 0x7fb3741011b0 2024-06-06T08:31:56.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb36a7fc640 1 -- 172.21.15.86:0/2650597226 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3700113a0 con 0x7fb3741011b0 2024-06-06T08:31:56.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.316+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37410d7f0 con 0x7fb3741011b0 2024-06-06T08:31:56.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.317+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3740fc920 con 0x7fb3741011b0 2024-06-06T08:31:56.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.317+0000 7fb36a7fc640 1 -- 172.21.15.86:0/2650597226 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fb370012400 con 0x7fb3741011b0 2024-06-06T08:31:56.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.318+0000 7fb36a7fc640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb360065430 0x7fb3600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:31:56.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.318+0000 7fb36a7fc640 1 -- 172.21.15.86:0/2650597226 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb3700825e0 con 0x7fb3741011b0 2024-06-06T08:31:56.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.318+0000 7fb37924b640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb360065430 0x7fb3600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:31:56.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.318+0000 7fb37924b640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb360065430 0x7fb3600678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb364009f80 tx=0x7fb364035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:31:56.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.322+0000 7fb36a7fc640 1 -- 172.21.15.86:0/2650597226 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb370050b10 con 0x7fb3741011b0 2024-06-06T08:31:56.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:56 smithi154 ceph-mon[35428]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:56.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.517+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb3740fa9a0 con 0x7fb3741011b0 2024-06-06T08:31:56.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.517+0000 7fb36a7fc640 1 -- 172.21.15.86:0/2650597226 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb370054130 con 0x7fb3741011b0 2024-06-06T08:31:56.516 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:31:56.516 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:31:56.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb360065430 msgr2=0x7fb3600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:56.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb360065430 0x7fb3600678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb364009f80 tx=0x7fb364035040 comp rx=0 tx=0).stop 2024-06-06T08:31:56.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3741011b0 msgr2=0x7fb374113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:31:56.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3741011b0 0x7fb374113020 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb37000c830 tx=0x7fb370004430 comp rx=0 tx=0).stop 2024-06-06T08:31:56.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 shutdown_connections 2024-06-06T08:31:56.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb374101b50 0x7fb374113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:56.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb360065430 0x7fb3600678e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:56.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 --2- 172.21.15.86:0/2650597226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3741011b0 0x7fb374113020 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:31:56.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 >> 172.21.15.86:0/2650597226 conn(0x7fb3740f6d60 msgr2=0x7fb3740f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:31:56.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 shutdown_connections 2024-06-06T08:31:56.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:31:56.519+0000 7fb37bcb0640 1 -- 172.21.15.86:0/2650597226 wait complete. 2024-06-06T08:31:57.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:57 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2650597226' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:57.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:57 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2650597226' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:31:58.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:31:58 smithi086 ceph-mon[29114]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:31:58.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:31:58 smithi154 ceph-mon[35428]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:00.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:32:00.332 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:32:00.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:00 smithi086 ceph-mon[29114]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:00.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:00 smithi154 ceph-mon[35428]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:01.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.735+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/883819755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e38101b70 msgr2=0x7f8e380fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:01.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.735+0000 7f8e3ddeb640 1 --2- 172.21.15.86:0/883819755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e38101b70 0x7f8e380fb200 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8e200048d0 tx=0x7f8e20009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:01.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.735+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/883819755 shutdown_connections 2024-06-06T08:32:01.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.735+0000 7f8e3ddeb640 1 --2- 172.21.15.86:0/883819755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e38101b70 0x7f8e380fb200 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:01.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.735+0000 7f8e3ddeb640 1 --2- 172.21.15.86:0/883819755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e381011d0 0x7f8e381015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:01.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.735+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/883819755 >> 172.21.15.86:0/883819755 conn(0x7f8e380f6dc0 msgr2=0x7f8e380f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:01.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.735+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/883819755 shutdown_connections 2024-06-06T08:32:01.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.735+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/883819755 wait complete. 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.735+0000 7f8e3ddeb640 1 Processor -- start 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e3ddeb640 1 -- start start 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e3ddeb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e381011d0 0x7f8e38113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e3ddeb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e38101b70 0x7f8e38113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e3ddeb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e38113b40 con 0x7f8e381011d0 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e3ddeb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3810cf80 con 0x7f8e38101b70 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e377fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e381011d0 0x7f8e38113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e377fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e381011d0 0x7f8e38113000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49054/0 (socket says 172.21.15.86:49054) 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e377fe640 1 -- 172.21.15.86:0/917945724 learned_addr learned my addr 172.21.15.86:0/917945724 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e36ffd640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e38101b70 0x7f8e38113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e377fe640 1 -- 172.21.15.86:0/917945724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e38101b70 msgr2=0x7f8e38113540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e377fe640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e38101b70 0x7f8e38113540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e377fe640 1 -- 172.21.15.86:0/917945724 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e20005040 con 0x7f8e381011d0 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.736+0000 7f8e377fe640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e381011d0 0x7f8e38113000 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e28002a50 tx=0x7f8e28004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.737+0000 7f8e2ffff640 1 -- 172.21.15.86:0/917945724 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e28007c30 con 0x7f8e381011d0 2024-06-06T08:32:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.737+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3810d260 con 0x7f8e381011d0 2024-06-06T08:32:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.737+0000 7f8e2ffff640 1 -- 172.21.15.86:0/917945724 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e2800f040 con 0x7f8e381011d0 2024-06-06T08:32:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.737+0000 7f8e2ffff640 1 -- 172.21.15.86:0/917945724 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e2800dad0 con 0x7f8e381011d0 2024-06-06T08:32:01.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.737+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3810d7d0 con 0x7f8e381011d0 2024-06-06T08:32:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.738+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e380fc940 con 0x7f8e381011d0 2024-06-06T08:32:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.738+0000 7f8e2ffff640 1 -- 172.21.15.86:0/917945724 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f8e2800dd40 con 0x7f8e381011d0 2024-06-06T08:32:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.738+0000 7f8e2ffff640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8e24065430 0x7f8e240678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.739+0000 7f8e2ffff640 1 -- 172.21.15.86:0/917945724 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8e28081c90 con 0x7f8e381011d0 2024-06-06T08:32:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.739+0000 7f8e36ffd640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8e24065430 0x7f8e240678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:01.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.739+0000 7f8e36ffd640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8e24065430 0x7f8e240678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009f80 tx=0x7f8e20035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:01.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.743+0000 7f8e2ffff640 1 -- 172.21.15.86:0/917945724 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8e2804cdf0 con 0x7f8e381011d0 2024-06-06T08:32:01.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.939+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8e38108330 con 0x7f8e381011d0 2024-06-06T08:32:01.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.939+0000 7f8e2ffff640 1 -- 172.21.15.86:0/917945724 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8e280501f0 con 0x7f8e381011d0 2024-06-06T08:32:01.939 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:32:01.939 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8e24065430 msgr2=0x7f8e240678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8e24065430 0x7f8e240678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009f80 tx=0x7f8e20035040 comp rx=0 tx=0).stop 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e381011d0 msgr2=0x7f8e38113000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e381011d0 0x7f8e38113000 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e28002a50 tx=0x7f8e28004600 comp rx=0 tx=0).stop 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 shutdown_connections 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e38101b70 0x7f8e38113540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f8e24065430 0x7f8e240678e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 --2- 172.21.15.86:0/917945724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e381011d0 0x7f8e38113000 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 >> 172.21.15.86:0/917945724 conn(0x7f8e380f6dc0 msgr2=0x7f8e380f7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 shutdown_connections 2024-06-06T08:32:01.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:01.941+0000 7f8e3ddeb640 1 -- 172.21.15.86:0/917945724 wait complete. 2024-06-06T08:32:02.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:02 smithi086 ceph-mon[29114]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:02.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:02 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/917945724' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:02.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:02 smithi154 ceph-mon[35428]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:02.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:02 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/917945724' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:04.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:04 smithi086 ceph-mon[29114]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:04.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:04 smithi154 ceph-mon[35428]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:05.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:32:05.876 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:32:06.307 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:06 smithi086 ceph-mon[29114]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:06.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:06 smithi154 ceph-mon[35428]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:07.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.298+0000 7fd60f589640 1 -- 172.21.15.86:0/131529319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6081022c0 msgr2=0x7fd608102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:07.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.298+0000 7fd60f589640 1 --2- 172.21.15.86:0/131529319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6081022c0 0x7fd608102690 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd6000048d0 tx=0x7fd600009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:07.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.298+0000 7fd60f589640 1 -- 172.21.15.86:0/131529319 shutdown_connections 2024-06-06T08:32:07.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.298+0000 7fd60f589640 1 --2- 172.21.15.86:0/131529319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd608102c60 0x7fd6080fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:07.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.298+0000 7fd60f589640 1 --2- 172.21.15.86:0/131529319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6081022c0 0x7fd608102690 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:07.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.298+0000 7fd60f589640 1 -- 172.21.15.86:0/131529319 >> 172.21.15.86:0/131529319 conn(0x7fd6080f7e50 msgr2=0x7fd6080fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:07.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.298+0000 7fd60f589640 1 -- 172.21.15.86:0/131529319 shutdown_connections 2024-06-06T08:32:07.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.298+0000 7fd60f589640 1 -- 172.21.15.86:0/131529319 wait complete. 2024-06-06T08:32:07.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.299+0000 7fd60f589640 1 Processor -- start 2024-06-06T08:32:07.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.299+0000 7fd60f589640 1 -- start start 2024-06-06T08:32:07.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.299+0000 7fd60f589640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6081022c0 0x7fd6081133d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:07.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.299+0000 7fd60f589640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd608102c60 0x7fd608113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.299+0000 7fd60f589640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd608113fa0 con 0x7fd608102c60 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.299+0000 7fd60f589640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60810d350 con 0x7fd6081022c0 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.299+0000 7fd60cb24640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd608102c60 0x7fd608113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.299+0000 7fd60cb24640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd608102c60 0x7fd608113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34770/0 (socket says 172.21.15.86:34770) 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.299+0000 7fd60cb24640 1 -- 172.21.15.86:0/1370565123 learned_addr learned my addr 172.21.15.86:0/1370565123 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd60d325640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6081022c0 0x7fd6081133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd60cb24640 1 -- 172.21.15.86:0/1370565123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6081022c0 msgr2=0x7fd6081133d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd60cb24640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6081022c0 0x7fd6081133d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd60cb24640 1 -- 172.21.15.86:0/1370565123 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd600005040 con 0x7fd608102c60 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd60cb24640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd608102c60 0x7fd608113910 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b790 tx=0x7fd5fc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd605ffb640 1 -- 172.21.15.86:0/1370565123 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5fc00c700 con 0x7fd608102c60 2024-06-06T08:32:07.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd605ffb640 1 -- 172.21.15.86:0/1370565123 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5fc00cd30 con 0x7fd608102c60 2024-06-06T08:32:07.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60810d630 con 0x7fd608102c60 2024-06-06T08:32:07.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd605ffb640 1 -- 172.21.15.86:0/1370565123 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5fc013370 con 0x7fd608102c60 2024-06-06T08:32:07.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.300+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60810db50 con 0x7fd608102c60 2024-06-06T08:32:07.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.301+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6080fda30 con 0x7fd608102c60 2024-06-06T08:32:07.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.301+0000 7fd605ffb640 1 -- 172.21.15.86:0/1370565123 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd5fc012070 con 0x7fd608102c60 2024-06-06T08:32:07.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.302+0000 7fd605ffb640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd5f0065430 0x7fd5f00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:07.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.302+0000 7fd605ffb640 1 -- 172.21.15.86:0/1370565123 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd5fc081fe0 con 0x7fd608102c60 2024-06-06T08:32:07.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.302+0000 7fd60d325640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd5f0065430 0x7fd5f00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:07.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.302+0000 7fd60d325640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd5f0065430 0x7fd5f00678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6000047f0 tx=0x7fd600009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:07.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.306+0000 7fd605ffb640 1 -- 172.21.15.86:0/1370565123 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd5fc050510 con 0x7fd608102c60 2024-06-06T08:32:07.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.504+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd6081094e0 con 0x7fd608102c60 2024-06-06T08:32:07.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.504+0000 7fd605ffb640 1 -- 172.21.15.86:0/1370565123 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd5fc04ee60 con 0x7fd608102c60 2024-06-06T08:32:07.504 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:32:07.504 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:32:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.506+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd5f0065430 msgr2=0x7fd5f00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.506+0000 7fd60f589640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd5f0065430 0x7fd5f00678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6000047f0 tx=0x7fd600009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.506+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd608102c60 msgr2=0x7fd608113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.506+0000 7fd60f589640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd608102c60 0x7fd608113910 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b790 tx=0x7fd5fc0091d0 comp rx=0 tx=0).stop 2024-06-06T08:32:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.506+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 shutdown_connections 2024-06-06T08:32:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.506+0000 7fd60f589640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd5f0065430 0x7fd5f00678e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.507+0000 7fd60f589640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd608102c60 0x7fd608113910 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:07.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.507+0000 7fd60f589640 1 --2- 172.21.15.86:0/1370565123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6081022c0 0x7fd6081133d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:07.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.507+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 >> 172.21.15.86:0/1370565123 conn(0x7fd6080f7e50 msgr2=0x7fd6080f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:07.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.507+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 shutdown_connections 2024-06-06T08:32:07.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:07.507+0000 7fd60f589640 1 -- 172.21.15.86:0/1370565123 wait complete. 2024-06-06T08:32:08.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:08 smithi086 ceph-mon[29114]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:08.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:08 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1370565123' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:08.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:08 smithi154 ceph-mon[35428]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:08.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:08 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1370565123' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:10.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:10 smithi086 ceph-mon[29114]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:10.337 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:10 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:32:10.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:10 smithi154 ceph-mon[35428]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:10.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:10 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:32:11.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:32:11.289 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:32:12.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:12 smithi086 ceph-mon[29114]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:12.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:12 smithi154 ceph-mon[35428]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:12.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 -- 172.21.15.86:0/2555491709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0141022a0 msgr2=0x7fd014102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:12.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 --2- 172.21.15.86:0/2555491709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0141022a0 0x7fd014102670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd004009c50 tx=0x7fd004004d10 comp rx=0 tx=0).stop 2024-06-06T08:32:12.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 -- 172.21.15.86:0/2555491709 shutdown_connections 2024-06-06T08:32:12.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 --2- 172.21.15.86:0/2555491709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd014102c40 0x7fd0140fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:12.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 --2- 172.21.15.86:0/2555491709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0141022a0 0x7fd014102670 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:12.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 -- 172.21.15.86:0/2555491709 >> 172.21.15.86:0/2555491709 conn(0x7fd0140f7e50 msgr2=0x7fd0140fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:12.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 -- 172.21.15.86:0/2555491709 shutdown_connections 2024-06-06T08:32:12.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 -- 172.21.15.86:0/2555491709 wait complete. 2024-06-06T08:32:12.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 Processor -- start 2024-06-06T08:32:12.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.795+0000 7fd019c5c640 1 -- start start 2024-06-06T08:32:12.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd019c5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0141022a0 0x7fd014113060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:12.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd019c5c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd014102c40 0x7fd0141135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:12.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd019c5c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd014113c30 con 0x7fd014102c40 2024-06-06T08:32:12.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd019c5c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01410cfe0 con 0x7fd0141022a0 2024-06-06T08:32:12.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd012ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd014102c40 0x7fd0141135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:12.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd012ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd014102c40 0x7fd0141135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34776/0 (socket says 172.21.15.86:34776) 2024-06-06T08:32:12.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd012ffd640 1 -- 172.21.15.86:0/1218066866 learned_addr learned my addr 172.21.15.86:0/1218066866 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:32:12.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd0137fe640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0141022a0 0x7fd014113060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:12.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd012ffd640 1 -- 172.21.15.86:0/1218066866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0141022a0 msgr2=0x7fd014113060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:12.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd012ffd640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0141022a0 0x7fd014113060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:12.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd012ffd640 1 -- 172.21.15.86:0/1218066866 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0040098b0 con 0x7fd014102c40 2024-06-06T08:32:12.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd0137fe640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0141022a0 0x7fd014113060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T08:32:12.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd012ffd640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd014102c40 0x7fd0141135a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00b790 tx=0x7fcffc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:12.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd00bfff640 1 -- 172.21.15.86:0/1218066866 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcffc00c700 con 0x7fd014102c40 2024-06-06T08:32:12.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd00bfff640 1 -- 172.21.15.86:0/1218066866 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcffc00cd30 con 0x7fd014102c40 2024-06-06T08:32:12.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01410d2c0 con 0x7fd014102c40 2024-06-06T08:32:12.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd00bfff640 1 -- 172.21.15.86:0/1218066866 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcffc013370 con 0x7fd014102c40 2024-06-06T08:32:12.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.796+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01410d7e0 con 0x7fd014102c40 2024-06-06T08:32:12.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.797+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0140fda10 con 0x7fd014102c40 2024-06-06T08:32:12.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.798+0000 7fd00bfff640 1 -- 172.21.15.86:0/1218066866 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fcffc012070 con 0x7fd014102c40 2024-06-06T08:32:12.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.798+0000 7fd00bfff640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd000065430 0x7fd0000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:12.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.798+0000 7fd00bfff640 1 -- 172.21.15.86:0/1218066866 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcffc082100 con 0x7fd014102c40 2024-06-06T08:32:12.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.798+0000 7fd0137fe640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd000065430 0x7fd0000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:12.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.798+0000 7fd0137fe640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd000065430 0x7fd0000678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd004003400 tx=0x7fd004004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:12.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:12.802+0000 7fd00bfff640 1 -- 172.21.15.86:0/1218066866 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcffc050620 con 0x7fd014102c40 2024-06-06T08:32:12.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.000+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd014109490 con 0x7fd014102c40 2024-06-06T08:32:12.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.001+0000 7fd00bfff640 1 -- 172.21.15.86:0/1218066866 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcffc053c40 con 0x7fd014102c40 2024-06-06T08:32:13.000 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:32:13.000 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:32:13.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd000065430 msgr2=0x7fd0000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd000065430 0x7fd0000678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd004003400 tx=0x7fd004004d10 comp rx=0 tx=0).stop 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd014102c40 msgr2=0x7fd0141135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd014102c40 0x7fd0141135a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00b790 tx=0x7fcffc0091d0 comp rx=0 tx=0).stop 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 shutdown_connections 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd000065430 0x7fd0000678e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd014102c40 0x7fd0141135a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 --2- 172.21.15.86:0/1218066866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0141022a0 0x7fd014113060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 >> 172.21.15.86:0/1218066866 conn(0x7fd0140f7e50 msgr2=0x7fd0140f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 shutdown_connections 2024-06-06T08:32:13.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:13.003+0000 7fd019c5c640 1 -- 172.21.15.86:0/1218066866 wait complete. 2024-06-06T08:32:13.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:13 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1218066866' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:13.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:13 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1218066866' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:14.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:14 smithi086 ceph-mon[29114]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:14.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:14 smithi154 ceph-mon[35428]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:16.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:16 smithi086 ceph-mon[29114]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:16.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:16 smithi154 ceph-mon[35428]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:16.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:32:16.930 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:32:18.188 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:18 smithi086 ceph-mon[29114]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:18.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.400+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1639281208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 msgr2=0x7f71b41027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:18.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.400+0000 7f71bb7b9640 1 --2- 172.21.15.86:0/1639281208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 0x7f71b41027b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f71b00048d0 tx=0x7f71b0009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:18.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1639281208 shutdown_connections 2024-06-06T08:32:18.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 --2- 172.21.15.86:0/1639281208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b40faa50 0x7f71b40faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:18.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 --2- 172.21.15.86:0/1639281208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 0x7f71b41027b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:18.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1639281208 >> 172.21.15.86:0/1639281208 conn(0x7f71b4070ab0 msgr2=0x7f71b4070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:18.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1639281208 shutdown_connections 2024-06-06T08:32:18.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1639281208 wait complete. 2024-06-06T08:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 Processor -- start 2024-06-06T08:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 -- start start 2024-06-06T08:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b40faa50 0x7f71b4115340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 0x7f71b4115880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4115f10 con 0x7f71b41023e0 2024-06-06T08:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71bb7b9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b410f310 con 0x7f71b40faa50 2024-06-06T08:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71b8d54640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 0x7f71b4115880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.401+0000 7f71b8d54640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 0x7f71b4115880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40880/0 (socket says 172.21.15.86:40880) 2024-06-06T08:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71b8d54640 1 -- 172.21.15.86:0/1729736549 learned_addr learned my addr 172.21.15.86:0/1729736549 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:32:18.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71b8d54640 1 -- 172.21.15.86:0/1729736549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b40faa50 msgr2=0x7f71b4115340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:18.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71b9555640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b40faa50 0x7f71b4115340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:18.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71b8d54640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b40faa50 0x7f71b4115340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:18.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71b8d54640 1 -- 172.21.15.86:0/1729736549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b0005040 con 0x7f71b41023e0 2024-06-06T08:32:18.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71b8d54640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 0x7f71b4115880 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f71a400b790 tx=0x7f71a40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:18.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71a9ffb640 1 -- 172.21.15.86:0/1729736549 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71a400c700 con 0x7f71b41023e0 2024-06-06T08:32:18.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71a9ffb640 1 -- 172.21.15.86:0/1729736549 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71a400cd30 con 0x7f71b41023e0 2024-06-06T08:32:18.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b410f5f0 con 0x7f71b41023e0 2024-06-06T08:32:18.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71a9ffb640 1 -- 172.21.15.86:0/1729736549 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71a4013370 con 0x7f71b41023e0 2024-06-06T08:32:18.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.402+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b410fb10 con 0x7f71b41023e0 2024-06-06T08:32:18.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.403+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b4065550 con 0x7f71b41023e0 2024-06-06T08:32:18.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.404+0000 7f71a9ffb640 1 -- 172.21.15.86:0/1729736549 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f71a4012070 con 0x7f71b41023e0 2024-06-06T08:32:18.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.404+0000 7f71a9ffb640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f71a0065430 0x7f71a00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:18.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.404+0000 7f71a9ffb640 1 -- 172.21.15.86:0/1729736549 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f71a4081e60 con 0x7f71b41023e0 2024-06-06T08:32:18.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.404+0000 7f71b9555640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f71a0065430 0x7f71a00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:18.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.404+0000 7f71b9555640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f71a0065430 0x7f71a00678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f71b0007ac0 tx=0x7f71b0009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:18.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.408+0000 7f71a9ffb640 1 -- 172.21.15.86:0/1729736549 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f71a4050310 con 0x7f71b41023e0 2024-06-06T08:32:18.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:18 smithi154 ceph-mon[35428]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:18.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.607+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f71b40749d0 con 0x7f71b41023e0 2024-06-06T08:32:18.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.607+0000 7f71a9ffb640 1 -- 172.21.15.86:0/1729736549 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f71a4053930 con 0x7f71b41023e0 2024-06-06T08:32:18.606 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:32:18.606 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:32:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f71a0065430 msgr2=0x7f71a00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f71a0065430 0x7f71a00678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f71b0007ac0 tx=0x7f71b0009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 msgr2=0x7f71b4115880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 0x7f71b4115880 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f71a400b790 tx=0x7f71a40091d0 comp rx=0 tx=0).stop 2024-06-06T08:32:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 shutdown_connections 2024-06-06T08:32:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7f71a0065430 0x7f71a00678e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b41023e0 0x7f71b4115880 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 --2- 172.21.15.86:0/1729736549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b40faa50 0x7f71b4115340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:18.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 >> 172.21.15.86:0/1729736549 conn(0x7f71b4070ab0 msgr2=0x7f71b40fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:18.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 shutdown_connections 2024-06-06T08:32:18.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:18.609+0000 7f71bb7b9640 1 -- 172.21.15.86:0/1729736549 wait complete. 2024-06-06T08:32:19.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:19 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/1729736549' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:19.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:19 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/1729736549' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:20.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:20 smithi086 ceph-mon[29114]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:20.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:20 smithi154 ceph-mon[35428]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:22.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:32:22.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:22 smithi154 ceph-mon[35428]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:22.500 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:32:22.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:22 smithi086 ceph-mon[29114]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:23.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 -- 172.21.15.86:0/957473787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fdc60 msgr2=0x7fd674106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:23.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 --2- 172.21.15.86:0/957473787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fdc60 0x7fd674106330 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd668004900 tx=0x7fd668009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:23.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 -- 172.21.15.86:0/957473787 shutdown_connections 2024-06-06T08:32:23.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 --2- 172.21.15.86:0/957473787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fdc60 0x7fd674106330 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 --2- 172.21.15.86:0/957473787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740fd2c0 0x7fd6740fd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 -- 172.21.15.86:0/957473787 >> 172.21.15.86:0/957473787 conn(0x7fd6740f6d60 msgr2=0x7fd6740f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 -- 172.21.15.86:0/957473787 shutdown_connections 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 -- 172.21.15.86:0/957473787 wait complete. 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 Processor -- start 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.879+0000 7fd679d6c640 1 -- start start 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd679d6c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fd2c0 0x7fd674113140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd679d6c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740fdc60 0x7fd674113680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd679d6c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd674113d10 con 0x7fd6740fd2c0 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd679d6c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67410d0c0 con 0x7fd6740fdc60 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd6737fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fd2c0 0x7fd674113140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd6737fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fd2c0 0x7fd674113140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40900/0 (socket says 172.21.15.86:40900) 2024-06-06T08:32:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd6737fe640 1 -- 172.21.15.86:0/2116031139 learned_addr learned my addr 172.21.15.86:0/2116031139 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd672ffd640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740fdc60 0x7fd674113680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd6737fe640 1 -- 172.21.15.86:0/2116031139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740fdc60 msgr2=0x7fd674113680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd6737fe640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740fdc60 0x7fd674113680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd6737fe640 1 -- 172.21.15.86:0/2116031139 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd668005040 con 0x7fd6740fd2c0 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd6737fe640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fd2c0 0x7fd674113140 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd66400b790 tx=0x7fd6640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd65bfff640 1 -- 172.21.15.86:0/2116031139 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd66400c700 con 0x7fd6740fd2c0 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd65bfff640 1 -- 172.21.15.86:0/2116031139 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd66400cd30 con 0x7fd6740fd2c0 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67410d3a0 con 0x7fd6740fd2c0 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd65bfff640 1 -- 172.21.15.86:0/2116031139 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd664013370 con 0x7fd6740fd2c0 2024-06-06T08:32:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.880+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67410d8c0 con 0x7fd6740fd2c0 2024-06-06T08:32:23.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.881+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd674066010 con 0x7fd6740fd2c0 2024-06-06T08:32:23.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.882+0000 7fd65bfff640 1 -- 172.21.15.86:0/2116031139 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd664012070 con 0x7fd6740fd2c0 2024-06-06T08:32:23.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.882+0000 7fd65bfff640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd65c065430 0x7fd65c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:23.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.882+0000 7fd65bfff640 1 -- 172.21.15.86:0/2116031139 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd664081e60 con 0x7fd6740fd2c0 2024-06-06T08:32:23.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.882+0000 7fd672ffd640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd65c065430 0x7fd65c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:23.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.883+0000 7fd672ffd640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd65c065430 0x7fd65c0678e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd668000c00 tx=0x7fd668009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:23.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:23.886+0000 7fd65bfff640 1 -- 172.21.15.86:0/2116031139 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd664050310 con 0x7fd6740fd2c0 2024-06-06T08:32:24.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.084+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd67404b4b0 con 0x7fd6740fd2c0 2024-06-06T08:32:24.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.084+0000 7fd65bfff640 1 -- 172.21.15.86:0/2116031139 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd664053930 con 0x7fd6740fd2c0 2024-06-06T08:32:24.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:32:24.084 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:32:24.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.086+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd65c065430 msgr2=0x7fd65c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:24.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.086+0000 7fd679d6c640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd65c065430 0x7fd65c0678e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd668000c00 tx=0x7fd668009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.086+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fd2c0 msgr2=0x7fd674113140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.086+0000 7fd679d6c640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fd2c0 0x7fd674113140 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd66400b790 tx=0x7fd6640091d0 comp rx=0 tx=0).stop 2024-06-06T08:32:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.086+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 shutdown_connections 2024-06-06T08:32:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.086+0000 7fd679d6c640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740fdc60 0x7fd674113680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.086+0000 7fd679d6c640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fd65c065430 0x7fd65c0678e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.086+0000 7fd679d6c640 1 --2- 172.21.15.86:0/2116031139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6740fd2c0 0x7fd674113140 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.086+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 >> 172.21.15.86:0/2116031139 conn(0x7fd6740f6d60 msgr2=0x7fd6740f8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.087+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 shutdown_connections 2024-06-06T08:32:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:24.087+0000 7fd679d6c640 1 -- 172.21.15.86:0/2116031139 wait complete. 2024-06-06T08:32:24.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:24 smithi086 ceph-mon[29114]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:24.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:24 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2116031139' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:24.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:24 smithi154 ceph-mon[35428]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:24.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:24 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2116031139' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:25.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:25 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:32:25.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:25 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:32:26.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:26 smithi154 ceph-mon[35428]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:26.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:26 smithi086 ceph-mon[29114]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:27.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:32:27.928 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:32:28.317 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:28 smithi086 ceph-mon[29114]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:28.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:28 smithi154 ceph-mon[35428]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.316+0000 7ff05e56b640 1 -- 172.21.15.86:0/1357406168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 msgr2=0x7ff0580fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.316+0000 7ff05e56b640 1 --2- 172.21.15.86:0/1357406168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 0x7ff0580fb200 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff0400048d0 tx=0x7ff040009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 -- 172.21.15.86:0/1357406168 shutdown_connections 2024-06-06T08:32:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 --2- 172.21.15.86:0/1357406168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 0x7ff0580fb200 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 --2- 172.21.15.86:0/1357406168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0581011d0 0x7ff0581015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 -- 172.21.15.86:0/1357406168 >> 172.21.15.86:0/1357406168 conn(0x7ff0580f6dc0 msgr2=0x7ff0580f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 -- 172.21.15.86:0/1357406168 shutdown_connections 2024-06-06T08:32:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 -- 172.21.15.86:0/1357406168 wait complete. 2024-06-06T08:32:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 Processor -- start 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 -- start start 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0581011d0 0x7ff058111f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 0x7ff058112460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058112a60 con 0x7ff058101b70 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.317+0000 7ff05e56b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05810bea0 con 0x7ff0581011d0 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff0577fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 0x7ff058112460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff0577fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 0x7ff058112460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60110/0 (socket says 172.21.15.86:60110) 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff0577fe640 1 -- 172.21.15.86:0/784072445 learned_addr learned my addr 172.21.15.86:0/784072445 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff057fff640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0581011d0 0x7ff058111f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff0577fe640 1 -- 172.21.15.86:0/784072445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0581011d0 msgr2=0x7ff058111f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:29.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff0577fe640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0581011d0 0x7ff058111f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:29.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff0577fe640 1 -- 172.21.15.86:0/784072445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff040005040 con 0x7ff058101b70 2024-06-06T08:32:29.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff0577fe640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 0x7ff058112460 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff0400043d0 tx=0x7ff040009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:29.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff054ff9640 1 -- 172.21.15.86:0/784072445 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff040038070 con 0x7ff058101b70 2024-06-06T08:32:29.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05810c120 con 0x7ff058101b70 2024-06-06T08:32:29.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff054ff9640 1 -- 172.21.15.86:0/784072445 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff040032970 con 0x7ff058101b70 2024-06-06T08:32:29.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff054ff9640 1 -- 172.21.15.86:0/784072445 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff04003da60 con 0x7ff058101b70 2024-06-06T08:32:29.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.318+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05810c660 con 0x7ff058101b70 2024-06-06T08:32:29.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.319+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0580fc940 con 0x7ff058101b70 2024-06-06T08:32:29.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.320+0000 7ff054ff9640 1 -- 172.21.15.86:0/784072445 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff040030020 con 0x7ff058101b70 2024-06-06T08:32:29.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.320+0000 7ff054ff9640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff044065430 0x7ff0440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:29.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.320+0000 7ff054ff9640 1 -- 172.21.15.86:0/784072445 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff0400a25c0 con 0x7ff058101b70 2024-06-06T08:32:29.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.320+0000 7ff057fff640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff044065430 0x7ff0440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:29.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.321+0000 7ff057fff640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff044065430 0x7ff0440678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff048004ed0 tx=0x7ff04800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:29.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.324+0000 7ff054ff9640 1 -- 172.21.15.86:0/784072445 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff040070a70 con 0x7ff058101b70 2024-06-06T08:32:29.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.526+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff0580fa9c0 con 0x7ff058101b70 2024-06-06T08:32:29.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.527+0000 7ff054ff9640 1 -- 172.21.15.86:0/784072445 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff040074090 con 0x7ff058101b70 2024-06-06T08:32:29.526 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:32:29.526 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff044065430 msgr2=0x7ff0440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff044065430 0x7ff0440678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff048004ed0 tx=0x7ff04800b040 comp rx=0 tx=0).stop 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 msgr2=0x7ff058112460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 0x7ff058112460 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff0400043d0 tx=0x7ff040009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 shutdown_connections 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7ff044065430 0x7ff0440678e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff058101b70 0x7ff058112460 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 --2- 172.21.15.86:0/784072445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0581011d0 0x7ff058111f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 >> 172.21.15.86:0/784072445 conn(0x7ff0580f6dc0 msgr2=0x7ff0580f7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:29.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 shutdown_connections 2024-06-06T08:32:29.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:29.529+0000 7ff05e56b640 1 -- 172.21.15.86:0/784072445 wait complete. 2024-06-06T08:32:30.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:30 smithi086 ceph-mon[29114]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:30.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:30 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/784072445' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:30.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:30 smithi154 ceph-mon[35428]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:30.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:30 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/784072445' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:32.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:32 smithi154 ceph-mon[35428]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:32.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:32 smithi086 ceph-mon[29114]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:33.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:32:33.399 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:32:34.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:34 smithi154 ceph-mon[35428]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:34.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:34 smithi086 ceph-mon[29114]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:34.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.753+0000 7fce31446640 1 -- 172.21.15.86:0/2571386813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 msgr2=0x7fce2c1062e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:34.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.753+0000 7fce31446640 1 --2- 172.21.15.86:0/2571386813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 0x7fce2c1062e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fce14004900 tx=0x7fce14009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:34.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.754+0000 7fce31446640 1 -- 172.21.15.86:0/2571386813 shutdown_connections 2024-06-06T08:32:34.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.754+0000 7fce31446640 1 --2- 172.21.15.86:0/2571386813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 0x7fce2c1062e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:34.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.754+0000 7fce31446640 1 --2- 172.21.15.86:0/2571386813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce2c0fd300 0x7fce2c0fd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:34.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.754+0000 7fce31446640 1 -- 172.21.15.86:0/2571386813 >> 172.21.15.86:0/2571386813 conn(0x7fce2c0f6d50 msgr2=0x7fce2c0f9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:34.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.754+0000 7fce31446640 1 -- 172.21.15.86:0/2571386813 shutdown_connections 2024-06-06T08:32:34.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.754+0000 7fce31446640 1 -- 172.21.15.86:0/2571386813 wait complete. 2024-06-06T08:32:34.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.754+0000 7fce31446640 1 Processor -- start 2024-06-06T08:32:34.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.754+0000 7fce31446640 1 -- start start 2024-06-06T08:32:34.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce31446640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce2c0fd300 0x7fce2c113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:34.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce31446640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 0x7fce2c1135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:34.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce31446640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c113c40 con 0x7fce2c0fdca0 2024-06-06T08:32:34.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce31446640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c10cff0 con 0x7fce2c0fd300 2024-06-06T08:32:34.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce2a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 0x7fce2c1135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:34.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce2a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 0x7fce2c1135b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60120/0 (socket says 172.21.15.86:60120) 2024-06-06T08:32:34.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce2a7fc640 1 -- 172.21.15.86:0/124787471 learned_addr learned my addr 172.21.15.86:0/124787471 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:32:34.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce2affd640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce2c0fd300 0x7fce2c113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:34.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce2a7fc640 1 -- 172.21.15.86:0/124787471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce2c0fd300 msgr2=0x7fce2c113070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:34.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce2a7fc640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce2c0fd300 0x7fce2c113070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:34.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce2a7fc640 1 -- 172.21.15.86:0/124787471 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce14005040 con 0x7fce2c0fdca0 2024-06-06T08:32:34.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce2a7fc640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 0x7fce2c1135b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fce14003080 tx=0x7fce14009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:34.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.755+0000 7fce237fe640 1 -- 172.21.15.86:0/124787471 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce14038070 con 0x7fce2c0fdca0 2024-06-06T08:32:34.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.756+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2c10d270 con 0x7fce2c0fdca0 2024-06-06T08:32:34.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.756+0000 7fce237fe640 1 -- 172.21.15.86:0/124787471 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce140338f0 con 0x7fce2c0fdca0 2024-06-06T08:32:34.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.756+0000 7fce237fe640 1 -- 172.21.15.86:0/124787471 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce1403c7e0 con 0x7fce2c0fdca0 2024-06-06T08:32:34.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.756+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2c10d730 con 0x7fce2c0fdca0 2024-06-06T08:32:34.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.757+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2c0660d0 con 0x7fce2c0fdca0 2024-06-06T08:32:34.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.757+0000 7fce237fe640 1 -- 172.21.15.86:0/124787471 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fce14030020 con 0x7fce2c0fdca0 2024-06-06T08:32:34.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.758+0000 7fce237fe640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fce18065430 0x7fce180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:34.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.758+0000 7fce2affd640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fce18065430 0x7fce180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:34.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.758+0000 7fce237fe640 1 -- 172.21.15.86:0/124787471 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fce140a2300 con 0x7fce2c0fdca0 2024-06-06T08:32:34.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.758+0000 7fce2affd640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fce18065430 0x7fce180678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fce24007580 tx=0x7fce24009780 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:34.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.762+0000 7fce237fe640 1 -- 172.21.15.86:0/124787471 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fce14070920 con 0x7fce2c0fdca0 2024-06-06T08:32:34.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.961+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce2c0fa990 con 0x7fce2c0fdca0 2024-06-06T08:32:34.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.961+0000 7fce237fe640 1 -- 172.21.15.86:0/124787471 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fce14073f40 con 0x7fce2c0fdca0 2024-06-06T08:32:34.962 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:32:34.962 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:32:34.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fce18065430 msgr2=0x7fce180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:34.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fce18065430 0x7fce180678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fce24007580 tx=0x7fce24009780 comp rx=0 tx=0).stop 2024-06-06T08:32:34.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 msgr2=0x7fce2c1135b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:34.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 0x7fce2c1135b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fce14003080 tx=0x7fce14009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:34.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 shutdown_connections 2024-06-06T08:32:34.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fce18065430 0x7fce180678e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:34.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce2c0fdca0 0x7fce2c1135b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:34.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 --2- 172.21.15.86:0/124787471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce2c0fd300 0x7fce2c113070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:34.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 >> 172.21.15.86:0/124787471 conn(0x7fce2c0f6d50 msgr2=0x7fce2c0f8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:34.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 shutdown_connections 2024-06-06T08:32:34.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:34.963+0000 7fce31446640 1 -- 172.21.15.86:0/124787471 wait complete. 2024-06-06T08:32:35.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:35 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/124787471' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:35.422 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:35 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/124787471' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:36.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:36 smithi154 ceph-mon[35428]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:36.584 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:36 smithi086 ceph-mon[29114]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:38.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:38 smithi154 ceph-mon[35428]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:38.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T08:32:38.474 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:38 smithi086 ceph-mon[29114]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:38.711 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/mon.smithi086/config 2024-06-06T08:32:40.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.117+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2321465229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 msgr2=0x7fb0740fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:40.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.117+0000 7fb07b5ed640 1 --2- 172.21.15.86:0/2321465229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 0x7fb0740fb200 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb0680048d0 tx=0x7fb068009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:40.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.117+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2321465229 shutdown_connections 2024-06-06T08:32:40.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.117+0000 7fb07b5ed640 1 --2- 172.21.15.86:0/2321465229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 0x7fb0740fb200 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:40.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.117+0000 7fb07b5ed640 1 --2- 172.21.15.86:0/2321465229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0741011d0 0x7fb0741015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:40.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.117+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2321465229 >> 172.21.15.86:0/2321465229 conn(0x7fb0740f6dc0 msgr2=0x7fb0740f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:40.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.117+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2321465229 shutdown_connections 2024-06-06T08:32:40.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.117+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2321465229 wait complete. 2024-06-06T08:32:40.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.117+0000 7fb07b5ed640 1 Processor -- start 2024-06-06T08:32:40.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb07b5ed640 1 -- start start 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb07b5ed640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0741011d0 0x7fb07410dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb07b5ed640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 0x7fb07410e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb07b5ed640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07410eae0 con 0x7fb074101b70 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb07b5ed640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07410ec50 con 0x7fb0741011d0 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb078b88640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 0x7fb07410e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb078b88640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 0x7fb07410e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51376/0 (socket says 172.21.15.86:51376) 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb078b88640 1 -- 172.21.15.86:0/2250369398 learned_addr learned my addr 172.21.15.86:0/2250369398 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb079389640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0741011d0 0x7fb07410dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb078b88640 1 -- 172.21.15.86:0/2250369398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0741011d0 msgr2=0x7fb07410dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb078b88640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0741011d0 0x7fb07410dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb078b88640 1 -- 172.21.15.86:0/2250369398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb068005040 con 0x7fb074101b70 2024-06-06T08:32:40.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb079389640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0741011d0 0x7fb07410dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T08:32:40.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.118+0000 7fb078b88640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 0x7fb07410e4e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb068003080 tx=0x7fb068009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:40.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.119+0000 7fb071ffb640 1 -- 172.21.15.86:0/2250369398 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb068038070 con 0x7fb074101b70 2024-06-06T08:32:40.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.119+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb074117010 con 0x7fb074101b70 2024-06-06T08:32:40.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.119+0000 7fb071ffb640 1 -- 172.21.15.86:0/2250369398 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb068008ee0 con 0x7fb074101b70 2024-06-06T08:32:40.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.119+0000 7fb071ffb640 1 -- 172.21.15.86:0/2250369398 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06803d870 con 0x7fb074101b70 2024-06-06T08:32:40.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.119+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb074117470 con 0x7fb074101b70 2024-06-06T08:32:40.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.120+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0740fc940 con 0x7fb074101b70 2024-06-06T08:32:40.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.120+0000 7fb071ffb640 1 -- 172.21.15.86:0/2250369398 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fb068030020 con 0x7fb074101b70 2024-06-06T08:32:40.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.121+0000 7fb071ffb640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb05c065430 0x7fb05c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T08:32:40.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.121+0000 7fb071ffb640 1 -- 172.21.15.86:0/2250369398 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb0680a2250 con 0x7fb074101b70 2024-06-06T08:32:40.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.121+0000 7fb079389640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb05c065430 0x7fb05c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T08:32:40.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.121+0000 7fb079389640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb05c065430 0x7fb05c0678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb06c007fd0 tx=0x7fb06c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T08:32:40.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.125+0000 7fb071ffb640 1 -- 172.21.15.86:0/2250369398 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb068070780 con 0x7fb074101b70 2024-06-06T08:32:40.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.335+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb0740fa9c0 con 0x7fb074101b70 2024-06-06T08:32:40.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:40 smithi086 ceph-mon[29114]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:40.334 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:40 smithi086 ceph-mon[29114]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:32:40.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.335+0000 7fb071ffb640 1 -- 172.21.15.86:0/2250369398 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb068073da0 con 0x7fb074101b70 2024-06-06T08:32:40.334 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-06T08:32:40.335 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T08:32:40.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb05c065430 msgr2=0x7fb05c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:40.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb05c065430 0x7fb05c0678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb06c007fd0 tx=0x7fb06c009040 comp rx=0 tx=0).stop 2024-06-06T08:32:40.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 msgr2=0x7fb07410e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T08:32:40.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 0x7fb07410e4e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb068003080 tx=0x7fb068009da0 comp rx=0 tx=0).stop 2024-06-06T08:32:40.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 shutdown_connections 2024-06-06T08:32:40.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:6800/3471926911,v1:172.21.15.86:6801/3471926911] conn(0x7fb05c065430 0x7fb05c0678e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:40.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb074101b70 0x7fb07410e4e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:40.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 --2- 172.21.15.86:0/2250369398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0741011d0 0x7fb07410dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T08:32:40.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 >> 172.21.15.86:0/2250369398 conn(0x7fb0740f6dc0 msgr2=0x7fb0740f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T08:32:40.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 shutdown_connections 2024-06-06T08:32:40.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-06-06T08:32:40.337+0000 7fb07b5ed640 1 -- 172.21.15.86:0/2250369398 wait complete. 2024-06-06T08:32:40.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:40 smithi154 ceph-mon[35428]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T08:32:40.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:40 smithi154 ceph-mon[35428]: from='mgr.14212 172.21.15.86:0/2096279248' entity='mgr.smithi086.mfkgze' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T08:32:40.843 INFO:tasks.cephadm:Teardown begin 2024-06-06T08:32:40.843 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-06T08:32:40.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T08:32:40.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T08:32:40.943 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-06T08:32:40.943 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-06T08:32:40.965 DEBUG:teuthology.orchestra.run.smithi154:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-06T08:32:41.001 INFO:tasks.cephadm:Stopping all daemons... 2024-06-06T08:32:41.002 INFO:tasks.cephadm.mon.smithi086:Stopping mon.smithi086... 2024-06-06T08:32:41.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi086 2024-06-06T08:32:41.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:41 smithi086 systemd[1]: Stopping Ceph mon.smithi086 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:32:41.416 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:41 smithi154 ceph-mon[35428]: from='client.? 172.21.15.86:0/2250369398' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:41.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:41 smithi086 ceph-mon[29114]: from='client.? 172.21.15.86:0/2250369398' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T08:32:41.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:41 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086[29110]: 2024-06-06T08:32:41.306+0000 7f9782dd7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi086 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T08:32:41.475 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:41 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086[29110]: 2024-06-06T08:32:41.306+0000 7f9782dd7640 -1 mon.smithi086@0(leader) e2 *** Got Signal Terminated *** 2024-06-06T08:32:41.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:41 smithi086 podman[80429]: 2024-06-06 08:32:41.510000099 +0000 UTC m=+0.311893210 container died 7a051db34bc21836df291f9223fbe26ec42d4a988dcdc6dbedfa4f4772b981c4 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-06-06T08:32:42.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:41 smithi086 podman[80429]: 2024-06-06 08:32:41.942003734 +0000 UTC m=+0.743896845 container cleanup 7a051db34bc21836df291f9223fbe26ec42d4a988dcdc6dbedfa4f4772b981c4 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-06-06T08:32:42.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:41 smithi086 bash[80429]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086 2024-06-06T08:32:42.347 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Jun 06 08:32:42 smithi086 podman[80518]: 2024-06-06 08:32:42.118498309 +0000 UTC m=+0.603193458 container remove 7a051db34bc21836df291f9223fbe26ec42d4a988dcdc6dbedfa4f4772b981c4 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi086, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3) 2024-06-06T08:32:42.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi086.service' 2024-06-06T08:32:42.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T08:32:42.409 INFO:tasks.cephadm.mon.smithi086:Stopped mon.smithi086 2024-06-06T08:32:42.409 INFO:tasks.cephadm.mon.smithi154:Stopping mon.smithi154... 2024-06-06T08:32:42.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi154 2024-06-06T08:32:42.734 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:42 smithi154 systemd[1]: Stopping Ceph mon.smithi154 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:32:43.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:42 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi154[35424]: 2024-06-06T08:32:42.735+0000 7fa940a58640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi154 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T08:32:43.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:42 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi154[35424]: 2024-06-06T08:32:42.735+0000 7fa940a58640 -1 mon.smithi154@1(peon) e2 *** Got Signal Terminated *** 2024-06-06T08:32:43.167 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:42 smithi154 podman[59248]: 2024-06-06 08:32:42.981259535 +0000 UTC m=+0.371874490 container died ff531b817756139da232e3b22f564bd46f04f859df25be2ccb39c6ad2be87866 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi154, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, ceph=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T08:32:43.666 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:43 smithi154 podman[59248]: 2024-06-06 08:32:43.444200806 +0000 UTC m=+0.834815722 container cleanup ff531b817756139da232e3b22f564bd46f04f859df25be2ccb39c6ad2be87866 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi154, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-06T08:32:43.667 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jun 06 08:32:43 smithi154 bash[59248]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-mon-smithi154 2024-06-06T08:32:43.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@mon.smithi154.service' 2024-06-06T08:32:43.924 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T08:32:43.924 INFO:tasks.cephadm.mon.smithi154:Stopped mon.smithi154 2024-06-06T08:32:43.924 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-06T08:32:43.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.0 2024-06-06T08:32:44.084 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:43 smithi086 systemd[1]: Stopping Ceph osd.0 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:32:44.584 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:44 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0[47081]: 2024-06-06T08:32:44.237+0000 7f3fd36f5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T08:32:44.584 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:44 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0[47081]: 2024-06-06T08:32:44.237+0000 7f3fd36f5640 -1 osd.0 40 *** Got signal Terminated *** 2024-06-06T08:32:44.584 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:44 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0[47081]: 2024-06-06T08:32:44.237+0000 7f3fd36f5640 -1 osd.0 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T08:32:49.834 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:49 smithi086 podman[80660]: 2024-06-06 08:32:49.564220617 +0000 UTC m=+5.429588223 container died 9d7881dc193ca39ba355b7f91a44c2fc41871f4bb5a825e2859444418c1fe88c (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892) 2024-06-06T08:32:50.212 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:50 smithi086 podman[80660]: 2024-06-06 08:32:50.004213645 +0000 UTC m=+5.869581271 container cleanup 9d7881dc193ca39ba355b7f91a44c2fc41871f4bb5a825e2859444418c1fe88c (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0, maintainer=Guillaume Abrioux , RELEASE=quincy-6ee6892, ceph=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-06T08:32:50.238 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:50 smithi086 bash[80660]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0 2024-06-06T08:32:50.584 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:50 smithi086 podman[80678]: 2024-06-06 08:32:50.21383719 +0000 UTC m=+0.645430127 container remove 9d7881dc193ca39ba355b7f91a44c2fc41871f4bb5a825e2859444418c1fe88c (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-6ee6892) 2024-06-06T08:32:50.890 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:50 smithi086 podman[80767]: 2024-06-06 08:32:50.623347632 +0000 UTC m=+0.106408700 container create d3357485625ffe1497680ae042849392d1ad20e2aed3f01498f68d564d4e66a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_CLEAN=True, RELEASE=quincy-6ee6892, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-06T08:32:50.891 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:50 smithi086 podman[80767]: 2024-06-06 08:32:50.534130266 +0000 UTC m=+0.017191317 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T08:32:50.891 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:50 smithi086 podman[80767]: 2024-06-06 08:32:50.834290168 +0000 UTC m=+0.317351200 container init d3357485625ffe1497680ae042849392d1ad20e2aed3f01498f68d564d4e66a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, ceph=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-06T08:32:50.891 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:50 smithi086 podman[80767]: 2024-06-06 08:32:50.838347676 +0000 UTC m=+0.321408706 container start d3357485625ffe1497680ae042849392d1ad20e2aed3f01498f68d564d4e66a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0) 2024-06-06T08:32:50.891 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:50 smithi086 podman[80767]: 2024-06-06 08:32:50.892105296 +0000 UTC m=+0.375166322 container attach d3357485625ffe1497680ae042849392d1ad20e2aed3f01498f68d564d4e66a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-06T08:32:51.319 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:51 smithi086 podman[80767]: 2024-06-06 08:32:51.085009161 +0000 UTC m=+0.568070190 container died d3357485625ffe1497680ae042849392d1ad20e2aed3f01498f68d564d4e66a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892) 2024-06-06T08:32:51.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.0.service' 2024-06-06T08:32:51.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:51 smithi086 podman[80791]: 2024-06-06 08:32:51.537052263 +0000 UTC m=+0.443490868 container remove d3357485625ffe1497680ae042849392d1ad20e2aed3f01498f68d564d4e66a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-0-deactivate, RELEASE=quincy-6ee6892, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-06-06T08:32:51.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:51 smithi086 systemd[1]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.0.service: Deactivated successfully. 2024-06-06T08:32:51.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:51 smithi086 systemd[1]: Stopped Ceph osd.0 for a8550dee-23dd-11ef-bc9b-c7b262605968. 2024-06-06T08:32:51.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 06 08:32:51 smithi086 systemd[1]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.0.service: Consumed 3.564s CPU time. 2024-06-06T08:32:51.597 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T08:32:51.597 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-06T08:32:51.597 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-06T08:32:51.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.1 2024-06-06T08:32:51.834 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:51 smithi086 systemd[1]: Stopping Ceph osd.1 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:32:52.334 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:51 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1[54687]: 2024-06-06T08:32:51.948+0000 7f04c14fe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T08:32:52.335 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:51 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1[54687]: 2024-06-06T08:32:51.948+0000 7f04c14fe640 -1 osd.1 40 *** Got signal Terminated *** 2024-06-06T08:32:52.335 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:51 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1[54687]: 2024-06-06T08:32:51.948+0000 7f04c14fe640 -1 osd.1 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T08:32:57.622 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:57 smithi086 podman[80871]: 2024-06-06 08:32:57.384560293 +0000 UTC m=+5.555147964 container died 9d403cb1c577f0e4c0f7f8fc128664d7d9621399e0db6d752fb0f4f272231fde (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892) 2024-06-06T08:32:57.964 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:57 smithi086 podman[80871]: 2024-06-06 08:32:57.764907529 +0000 UTC m=+5.935495209 container cleanup 9d403cb1c577f0e4c0f7f8fc128664d7d9621399e0db6d752fb0f4f272231fde (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, CEPH_POINT_RELEASE=, ceph=True) 2024-06-06T08:32:57.964 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:57 smithi086 bash[80871]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1 2024-06-06T08:32:58.290 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:57 smithi086 podman[80886]: 2024-06-06 08:32:57.966119205 +0000 UTC m=+0.582755692 container remove 9d403cb1c577f0e4c0f7f8fc128664d7d9621399e0db6d752fb0f4f272231fde (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892) 2024-06-06T08:32:58.542 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:58 smithi086 podman[80980]: 2024-06-06 08:32:58.194757945 +0000 UTC m=+0.017377798 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T08:32:58.542 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:58 smithi086 podman[80980]: 2024-06-06 08:32:58.300376568 +0000 UTC m=+0.122996403 container create 6ecb10d4f540ac1aec4fd2a34f451e3754e4811c67b52fa482741b6d719dba12 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3) 2024-06-06T08:32:58.542 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:58 smithi086 podman[80980]: 2024-06-06 08:32:58.494042402 +0000 UTC m=+0.316662254 container init 6ecb10d4f540ac1aec4fd2a34f451e3754e4811c67b52fa482741b6d719dba12 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6ee6892, org.label-schema.build-date=20240528, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-06T08:32:58.542 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:58 smithi086 podman[80980]: 2024-06-06 08:32:58.497233722 +0000 UTC m=+0.319853573 container start 6ecb10d4f540ac1aec4fd2a34f451e3754e4811c67b52fa482741b6d719dba12 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, io.buildah.version=1.35.3) 2024-06-06T08:32:58.542 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:58 smithi086 podman[80980]: 2024-06-06 08:32:58.544159062 +0000 UTC m=+0.366778923 container attach 6ecb10d4f540ac1aec4fd2a34f451e3754e4811c67b52fa482741b6d719dba12 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6ee6892, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-06-06T08:32:58.834 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 06 08:32:58 smithi086 podman[80980]: 2024-06-06 08:32:58.736439342 +0000 UTC m=+0.559059185 container died 6ecb10d4f540ac1aec4fd2a34f451e3754e4811c67b52fa482741b6d719dba12 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-06-06T08:32:59.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.1.service' 2024-06-06T08:32:59.429 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T08:32:59.429 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-06T08:32:59.429 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-06T08:32:59.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.2 2024-06-06T08:32:59.732 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:32:59 smithi086 systemd[1]: Stopping Ceph osd.2 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:33:00.084 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:32:59 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2[61965]: 2024-06-06T08:32:59.733+0000 7f1bb9c34640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T08:33:00.084 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:32:59 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2[61965]: 2024-06-06T08:32:59.733+0000 7f1bb9c34640 -1 osd.2 40 *** Got signal Terminated *** 2024-06-06T08:33:00.084 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:32:59 smithi086 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2[61965]: 2024-06-06T08:32:59.733+0000 7f1bb9c34640 -1 osd.2 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T08:33:05.515 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:05 smithi086 podman[81082]: 2024-06-06 08:33:05.197138686 +0000 UTC m=+5.539878046 container died be5f4f9b215daabda23f09c6e97c1359485708279e51ff4b14d4693240fb714b (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True) 2024-06-06T08:33:05.515 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:05 smithi086 podman[81082]: 2024-06-06 08:33:05.516802523 +0000 UTC m=+5.859541874 container cleanup be5f4f9b215daabda23f09c6e97c1359485708279e51ff4b14d4693240fb714b (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, RELEASE=quincy-6ee6892, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=) 2024-06-06T08:33:05.834 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:05 smithi086 bash[81082]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2 2024-06-06T08:33:05.834 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:05 smithi086 podman[81097]: 2024-06-06 08:33:05.73478646 +0000 UTC m=+0.539003115 container remove be5f4f9b215daabda23f09c6e97c1359485708279e51ff4b14d4693240fb714b (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-06-06T08:33:06.334 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:06 smithi086 podman[81190]: 2024-06-06 08:33:06.044005229 +0000 UTC m=+0.099483210 container create 70bc5da502ca61f70b9406cb202353a78e1ddeaebed99878f40d11414040bc39 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS) 2024-06-06T08:33:06.335 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:06 smithi086 podman[81190]: 2024-06-06 08:33:05.961514339 +0000 UTC m=+0.016992286 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T08:33:06.335 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:06 smithi086 podman[81190]: 2024-06-06 08:33:06.229565938 +0000 UTC m=+0.285043887 container init 70bc5da502ca61f70b9406cb202353a78e1ddeaebed99878f40d11414040bc39 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_CLEAN=True, RELEASE=quincy-6ee6892) 2024-06-06T08:33:06.335 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:06 smithi086 podman[81190]: 2024-06-06 08:33:06.232802704 +0000 UTC m=+0.288280644 container start 70bc5da502ca61f70b9406cb202353a78e1ddeaebed99878f40d11414040bc39 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-06T08:33:06.335 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:06 smithi086 podman[81190]: 2024-06-06 08:33:06.287656485 +0000 UTC m=+0.343134439 container attach 70bc5da502ca61f70b9406cb202353a78e1ddeaebed99878f40d11414040bc39 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2-deactivate, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-06T08:33:06.814 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:06 smithi086 podman[81190]: 2024-06-06 08:33:06.494092701 +0000 UTC m=+0.549570647 container died 70bc5da502ca61f70b9406cb202353a78e1ddeaebed99878f40d11414040bc39 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=quincy-6ee6892, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T08:33:07.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.2.service' 2024-06-06T08:33:07.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:07 smithi086 podman[81212]: 2024-06-06 08:33:07.024360164 +0000 UTC m=+0.520370425 container remove 70bc5da502ca61f70b9406cb202353a78e1ddeaebed99878f40d11414040bc39 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T08:33:07.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:07 smithi086 systemd[1]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.2.service: Deactivated successfully. 2024-06-06T08:33:07.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:07 smithi086 systemd[1]: Stopped Ceph osd.2 for a8550dee-23dd-11ef-bc9b-c7b262605968. 2024-06-06T08:33:07.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 06 08:33:07 smithi086 systemd[1]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.2.service: Consumed 3.438s CPU time. 2024-06-06T08:33:07.083 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T08:33:07.083 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-06T08:33:07.083 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-06T08:33:07.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.3 2024-06-06T08:33:07.377 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:07 smithi154 systemd[1]: Stopping Ceph osd.3 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:33:07.667 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:07 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3[40879]: 2024-06-06T08:33:07.378+0000 7fc9a104b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T08:33:07.667 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:07 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3[40879]: 2024-06-06T08:33:07.378+0000 7fc9a104b640 -1 osd.3 40 *** Got signal Terminated *** 2024-06-06T08:33:07.667 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:07 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3[40879]: 2024-06-06T08:33:07.378+0000 7fc9a104b640 -1 osd.3 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T08:33:12.416 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:12 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3[40879]: 2024-06-06T08:33:12.102+0000 7fc99e665640 -1 osd.3 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:45.806810+0000 front 2024-06-06T08:32:45.806776+0000 (oldest deadline 2024-06-06T08:33:11.706707+0000) 2024-06-06T08:33:13.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:13 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:13.046+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:13.167 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:12 smithi154 podman[59372]: 2024-06-06 08:33:12.90277018 +0000 UTC m=+5.601901334 container died 9d1b4f896864e17f5212da29c5ee2807410bb292a1f72ee21c9e5714710ba058 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-06T08:33:13.496 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:13 smithi154 podman[59372]: 2024-06-06 08:33:13.296214389 +0000 UTC m=+5.995345521 container cleanup 9d1b4f896864e17f5212da29c5ee2807410bb292a1f72ee21c9e5714710ba058 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-6ee6892, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-06T08:33:13.496 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:13 smithi154 bash[59372]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3 2024-06-06T08:33:13.810 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:13 smithi154 podman[59398]: 2024-06-06 08:33:13.498083216 +0000 UTC m=+0.597777450 container remove 9d1b4f896864e17f5212da29c5ee2807410bb292a1f72ee21c9e5714710ba058 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-6ee6892, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, CEPH_POINT_RELEASE=) 2024-06-06T08:33:14.116 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:13 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:13.811+0000 7fb89a77a640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:47.508590+0000 front 2024-06-06T08:32:47.508533+0000 (oldest deadline 2024-06-06T08:33:13.408423+0000) 2024-06-06T08:33:14.116 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:14 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:14.001+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:14.117 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:13 smithi154 podman[59494]: 2024-06-06 08:33:13.908072585 +0000 UTC m=+0.105784963 container create 1c7b96bf538cdff6e42d5cdd09a7deca3a12446c07c24dfafcf1492cee014f6c (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2) 2024-06-06T08:33:14.117 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:13 smithi154 podman[59494]: 2024-06-06 08:33:13.819881269 +0000 UTC m=+0.017593673 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T08:33:14.117 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:14 smithi154 podman[59494]: 2024-06-06 08:33:14.118672305 +0000 UTC m=+0.316384688 container init 1c7b96bf538cdff6e42d5cdd09a7deca3a12446c07c24dfafcf1492cee014f6c (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3-deactivate, RELEASE=quincy-6ee6892, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS) 2024-06-06T08:33:14.416 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:14 smithi154 podman[59494]: 2024-06-06 08:33:14.122623151 +0000 UTC m=+0.320335529 container start 1c7b96bf538cdff6e42d5cdd09a7deca3a12446c07c24dfafcf1492cee014f6c (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3-deactivate, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0) 2024-06-06T08:33:14.417 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:14 smithi154 podman[59494]: 2024-06-06 08:33:14.193771756 +0000 UTC m=+0.391484166 container attach 1c7b96bf538cdff6e42d5cdd09a7deca3a12446c07c24dfafcf1492cee014f6c (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3-deactivate, ceph=True, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD) 2024-06-06T08:33:14.417 INFO:journalctl@ceph.osd.3.smithi154.stdout:Jun 06 08:33:14 smithi154 podman[59494]: 2024-06-06 08:33:14.364661453 +0000 UTC m=+0.562373834 container died 1c7b96bf538cdff6e42d5cdd09a7deca3a12446c07c24dfafcf1492cee014f6c (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, ceph=True) 2024-06-06T08:33:14.917 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:14 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:14.797+0000 7fb89a77a640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:47.508590+0000 front 2024-06-06T08:32:47.508533+0000 (oldest deadline 2024-06-06T08:33:13.408423+0000) 2024-06-06T08:33:14.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.3.service' 2024-06-06T08:33:14.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T08:33:14.986 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-06T08:33:14.986 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-06T08:33:14.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.4 2024-06-06T08:33:15.307 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:15 smithi154 systemd[1]: Stopping Ceph osd.4 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:33:15.307 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:15 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:15.009+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:15.666 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:15 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:15.308+0000 7fb89e963640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T08:33:15.667 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:15 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:15.308+0000 7fb89e963640 -1 osd.4 40 *** Got signal Terminated *** 2024-06-06T08:33:15.667 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:15 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:15.308+0000 7fb89e963640 -1 osd.4 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T08:33:16.166 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:15 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:15.762+0000 7fb89a77a640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:47.508590+0000 front 2024-06-06T08:32:47.508533+0000 (oldest deadline 2024-06-06T08:33:13.408423+0000) 2024-06-06T08:33:16.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:15 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:15.972+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:16.916 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:16 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:16.755+0000 7fb89a77a640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:47.508590+0000 front 2024-06-06T08:32:47.508533+0000 (oldest deadline 2024-06-06T08:33:13.408423+0000) 2024-06-06T08:33:17.416 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:16 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:16.961+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:17.972 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:17 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:17.713+0000 7fb89a77a640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:47.508590+0000 front 2024-06-06T08:32:47.508533+0000 (oldest deadline 2024-06-06T08:33:13.408423+0000) 2024-06-06T08:33:18.391 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:17 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:17.973+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:19.166 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:18 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:18.733+0000 7fb89a77a640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:47.508590+0000 front 2024-06-06T08:32:47.508533+0000 (oldest deadline 2024-06-06T08:33:13.408423+0000) 2024-06-06T08:33:19.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:18 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:18.932+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:20.166 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:19 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:19.747+0000 7fb89a77a640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:47.508590+0000 front 2024-06-06T08:32:47.508533+0000 (oldest deadline 2024-06-06T08:33:13.408423+0000) 2024-06-06T08:33:20.167 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:19 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4[47685]: 2024-06-06T08:33:19.747+0000 7fb89a77a640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:53.408854+0000 front 2024-06-06T08:32:53.408871+0000 (oldest deadline 2024-06-06T08:33:19.308683+0000) 2024-06-06T08:33:20.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:19 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:19.906+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:20.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:19 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:19.906+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:56.616190+0000 front 2024-06-06T08:32:56.616108+0000 (oldest deadline 2024-06-06T08:33:19.515901+0000) 2024-06-06T08:33:20.973 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:20 smithi154 podman[59598]: 2024-06-06 08:33:20.670345389 +0000 UTC m=+5.456277083 container died 15c1e9349b4f6be303d5078d06dfb5d38334df72de4a65e88c52b2c9f8f97bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, RELEASE=quincy-6ee6892, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-06-06T08:33:20.974 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:20 smithi154 podman[59598]: 2024-06-06 08:33:20.975516704 +0000 UTC m=+5.761448399 container cleanup 15c1e9349b4f6be303d5078d06dfb5d38334df72de4a65e88c52b2c9f8f97bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528) 2024-06-06T08:33:20.974 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:20 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:20.882+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:20.974 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:20 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:20.882+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:56.616190+0000 front 2024-06-06T08:32:56.616108+0000 (oldest deadline 2024-06-06T08:33:19.515901+0000) 2024-06-06T08:33:21.416 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:20 smithi154 bash[59598]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4 2024-06-06T08:33:21.416 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:21 smithi154 podman[59618]: 2024-06-06 08:33:21.185173635 +0000 UTC m=+0.512475718 container remove 15c1e9349b4f6be303d5078d06dfb5d38334df72de4a65e88c52b2c9f8f97bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3) 2024-06-06T08:33:21.803 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:21 smithi154 podman[59706]: 2024-06-06 08:33:21.510942458 +0000 UTC m=+0.099533184 container create 2968eca33bea75f905182a6cbacd22b2317b563fb15c1ad1871dcb99d2c299e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, RELEASE=quincy-6ee6892, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-06T08:33:21.803 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:21 smithi154 podman[59706]: 2024-06-06 08:33:21.428354419 +0000 UTC m=+0.016945146 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T08:33:21.803 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:21 smithi154 podman[59706]: 2024-06-06 08:33:21.730266196 +0000 UTC m=+0.318856924 container init 2968eca33bea75f905182a6cbacd22b2317b563fb15c1ad1871dcb99d2c299e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-6ee6892, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-06T08:33:21.803 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:21 smithi154 podman[59706]: 2024-06-06 08:33:21.733490424 +0000 UTC m=+0.322081162 container start 2968eca33bea75f905182a6cbacd22b2317b563fb15c1ad1871dcb99d2c299e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-6ee6892) 2024-06-06T08:33:21.804 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:21 smithi154 podman[59706]: 2024-06-06 08:33:21.8053758 +0000 UTC m=+0.393966538 container attach 2968eca33bea75f905182a6cbacd22b2317b563fb15c1ad1871dcb99d2c299e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-06T08:33:22.167 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:21 smithi154 podman[59706]: 2024-06-06 08:33:21.981571818 +0000 UTC m=+0.570162555 container died 2968eca33bea75f905182a6cbacd22b2317b563fb15c1ad1871dcb99d2c299e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.35.3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux ) 2024-06-06T08:33:22.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:21 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:21.834+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:22.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:21 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:21.834+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:56.616190+0000 front 2024-06-06T08:32:56.616108+0000 (oldest deadline 2024-06-06T08:33:19.515901+0000) 2024-06-06T08:33:22.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.4.service' 2024-06-06T08:33:22.519 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:22 smithi154 podman[59729]: 2024-06-06 08:33:22.475492383 +0000 UTC m=+0.484625513 container remove 2968eca33bea75f905182a6cbacd22b2317b563fb15c1ad1871dcb99d2c299e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892, ceph=True) 2024-06-06T08:33:22.519 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:22 smithi154 systemd[1]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.4.service: Deactivated successfully. 2024-06-06T08:33:22.519 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:22 smithi154 systemd[1]: Stopped Ceph osd.4 for a8550dee-23dd-11ef-bc9b-c7b262605968. 2024-06-06T08:33:22.519 INFO:journalctl@ceph.osd.4.smithi154.stdout:Jun 06 08:33:22 smithi154 systemd[1]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.4.service: Consumed 2.761s CPU time. 2024-06-06T08:33:22.533 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T08:33:22.533 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-06T08:33:22.533 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-06T08:33:22.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.5 2024-06-06T08:33:22.816 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:22 smithi154 systemd[1]: Stopping Ceph osd.5 for a8550dee-23dd-11ef-bc9b-c7b262605968... 2024-06-06T08:33:23.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:22 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:22.817+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:23.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:22 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:22.817+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:56.616190+0000 front 2024-06-06T08:32:56.616108+0000 (oldest deadline 2024-06-06T08:33:19.515901+0000) 2024-06-06T08:33:23.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:22 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:22.879+0000 7fad50a0d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T08:33:23.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:22 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:22.879+0000 7fad50a0d640 -1 osd.5 40 *** Got signal Terminated *** 2024-06-06T08:33:23.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:22 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:22.879+0000 7fad50a0d640 -1 osd.5 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T08:33:24.166 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:23 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:23.828+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:24.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:23 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:23.828+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:56.616190+0000 front 2024-06-06T08:32:56.616108+0000 (oldest deadline 2024-06-06T08:33:19.515901+0000) 2024-06-06T08:33:25.166 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:24 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:24.854+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:25.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:24 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:24.854+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:56.616190+0000 front 2024-06-06T08:32:56.616108+0000 (oldest deadline 2024-06-06T08:33:19.515901+0000) 2024-06-06T08:33:26.166 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:25 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:25.831+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:26.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:25 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:25.831+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:56.616190+0000 front 2024-06-06T08:32:56.616108+0000 (oldest deadline 2024-06-06T08:33:19.515901+0000) 2024-06-06T08:33:26.916 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:26 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:26.794+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:26.917 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:26 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:26.794+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:56.616190+0000 front 2024-06-06T08:32:56.616108+0000 (oldest deadline 2024-06-06T08:33:19.515901+0000) 2024-06-06T08:33:28.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:27 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:27.767+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-06T08:32:46.615369+0000 front 2024-06-06T08:32:46.615407+0000 (oldest deadline 2024-06-06T08:33:12.515208+0000) 2024-06-06T08:33:28.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:27 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:27.767+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-06T08:32:56.616190+0000 front 2024-06-06T08:32:56.616108+0000 (oldest deadline 2024-06-06T08:33:19.515901+0000) 2024-06-06T08:33:28.167 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:27 smithi154 ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5[54355]: 2024-06-06T08:33:27.767+0000 7fad4c023640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-06T08:33:03.517139+0000 front 2024-06-06T08:33:03.517126+0000 (oldest deadline 2024-06-06T08:33:27.017088+0000) 2024-06-06T08:33:28.460 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:28 smithi154 podman[59808]: 2024-06-06 08:33:28.188228431 +0000 UTC m=+5.431787998 container died 6462108b8870a971d0c7555788f4a2b7b5a6a39342f59f7bd1fbe489d77ac75a (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-06T08:33:28.794 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:28 smithi154 podman[59808]: 2024-06-06 08:33:28.537130387 +0000 UTC m=+5.780689954 container cleanup 6462108b8870a971d0c7555788f4a2b7b5a6a39342f59f7bd1fbe489d77ac75a (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, RELEASE=quincy-6ee6892, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.35.3) 2024-06-06T08:33:28.794 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:28 smithi154 bash[59808]: ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5 2024-06-06T08:33:29.154 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:28 smithi154 podman[59825]: 2024-06-06 08:33:28.796382569 +0000 UTC m=+0.607069249 container remove 6462108b8870a971d0c7555788f4a2b7b5a6a39342f59f7bd1fbe489d77ac75a (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-06-06T08:33:29.414 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:29 smithi154 podman[59910]: 2024-06-06 08:33:29.15563234 +0000 UTC m=+0.111347609 container create 4910554e2491ddea5a9d6d028860662c689d4df88b271929ce876eee7d74c5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, RELEASE=quincy-6ee6892, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, ceph=True, org.label-schema.license=GPLv2) 2024-06-06T08:33:29.415 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:29 smithi154 podman[59910]: 2024-06-06 08:33:29.061604783 +0000 UTC m=+0.017320023 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T08:33:29.415 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:29 smithi154 podman[59910]: 2024-06-06 08:33:29.341519407 +0000 UTC m=+0.297234627 container init 4910554e2491ddea5a9d6d028860662c689d4df88b271929ce876eee7d74c5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-06T08:33:29.415 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:29 smithi154 podman[59910]: 2024-06-06 08:33:29.344886426 +0000 UTC m=+0.300601661 container start 4910554e2491ddea5a9d6d028860662c689d4df88b271929ce876eee7d74c5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T08:33:29.666 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:29 smithi154 podman[59910]: 2024-06-06 08:33:29.416398323 +0000 UTC m=+0.372113577 container attach 4910554e2491ddea5a9d6d028860662c689d4df88b271929ce876eee7d74c5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=quincy-6ee6892, ceph=True, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-06-06T08:33:29.667 INFO:journalctl@ceph.osd.5.smithi154.stdout:Jun 06 08:33:29 smithi154 podman[59910]: 2024-06-06 08:33:29.593146018 +0000 UTC m=+0.548861247 container died 4910554e2491ddea5a9d6d028860662c689d4df88b271929ce876eee7d74c5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-a8550dee-23dd-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True) 2024-06-06T08:33:30.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a8550dee-23dd-11ef-bc9b-c7b262605968@osd.5.service' 2024-06-06T08:33:30.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T08:33:30.223 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-06T08:33:30.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-06T08:33:38.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-06T08:33:43.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T08:33:43.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T08:33:43.275 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-06T08:33:43.277 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240/remote/smithi086/crash 2024-06-06T08:33:43.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/crash -- . 2024-06-06T08:33:43.334 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-06T08:33:43.335 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-06-06T08:33:43.337 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240/remote/smithi154/crash 2024-06-06T08:33:43.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/crash -- . 2024-06-06T08:33:43.375 INFO:teuthology.orchestra.run.smithi154.stderr:tar: /var/lib/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-06T08:33:43.375 INFO:teuthology.orchestra.run.smithi154.stderr:tar: Error is not recoverable: exiting now 2024-06-06T08:33:43.376 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-06T08:33:43.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-06T08:33:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:1717662167.3590875 mon.smithi086 (mon.0) 121 : cluster [WRN] Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-06T08:33:43.410 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-06T08:33:43.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[SEC\]' /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-06T08:33:43.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]' /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-06T08:33:43.554 INFO:teuthology.orchestra.run.smithi086.stdout:1717662227.7967122 mgr.smithi154.kxiapq (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents 2024-06-06T08:33:43.554 INFO:tasks.cephadm:Compressing logs... 2024-06-06T08:33:43.554 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-06T08:33:43.597 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-06T08:33:43.619 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.log 2024-06-06T08:33:43.619 INFO:teuthology.orchestra.run.smithi086.stderr:: No such file or directory 2024-06-06T08:33:43.619 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-06-06T08:33:43.620 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-06T08:33:43.620 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mgr.smithi086.mfkgze.log 2024-06-06T08:33:43.620 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-06-06T08:33:43.621 INFO:teuthology.orchestra.run.smithi086.stderr: 87.5% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-06T08:33:43.621 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-06-06T08:33:43.625 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mgr.smithi086.mfkgze.log: /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mon.smithi086.log 2024-06-06T08:33:43.625 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-06T08:33:43.625 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mon.smithi086.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-06T08:33:43.625 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-06T08:33:43.625 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-06T08:33:43.625 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-volume.log: 83.4% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-06T08:33:43.626 INFO:teuthology.orchestra.run.smithi154.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.log 2024-06-06T08:33:43.626 INFO:teuthology.orchestra.run.smithi154.stderr:: No such file or directory 2024-06-06T08:33:43.626 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-06T08:33:43.626 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-06-06T08:33:43.626 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-06-06T08:33:43.626 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-06-06T08:33:43.626 INFO:teuthology.orchestra.run.smithi154.stderr: 87.8% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-06T08:33:43.627 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mon.smithi154.log 2024-06-06T08:33:43.627 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.3.log: /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-06T08:33:43.627 INFO:teuthology.orchestra.run.smithi154.stderr: 90.3% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-06T08:33:43.627 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mon.smithi154.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mgr.smithi154.kxiapq.log 2024-06-06T08:33:43.627 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-06T08:33:43.627 INFO:teuthology.orchestra.run.smithi154.stderr: 81.7% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-06T08:33:43.627 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mgr.smithi154.kxiapq.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-06T08:33:43.627 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-06T08:33:43.629 INFO:teuthology.orchestra.run.smithi154.stderr: 92.4% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mgr.smithi154.kxiapq.log.gz 2024-06-06T08:33:43.631 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/cephadm.log: 93.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-06T08:33:43.632 INFO:teuthology.orchestra.run.smithi086.stderr: 90.9% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-06T08:33:43.634 INFO:teuthology.orchestra.run.smithi154.stderr: 90.9% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-06T08:33:43.666 INFO:teuthology.orchestra.run.smithi154.stderr: 93.0% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-06-06T08:33:43.669 INFO:teuthology.orchestra.run.smithi154.stderr: 93.0% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-06-06T08:33:43.677 INFO:teuthology.orchestra.run.smithi154.stderr: 92.4% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-06-06T08:33:43.680 INFO:teuthology.orchestra.run.smithi086.stderr: 92.8% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-06-06T08:33:43.702 INFO:teuthology.orchestra.run.smithi086.stderr: 93.1% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-06-06T08:33:43.707 INFO:teuthology.orchestra.run.smithi154.stderr: 92.8% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mon.smithi154.log.gz 2024-06-06T08:33:43.709 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-06-06T08:33:43.709 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m0.098s 2024-06-06T08:33:43.709 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m0.245s 2024-06-06T08:33:43.709 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m0.029s 2024-06-06T08:33:43.717 INFO:teuthology.orchestra.run.smithi086.stderr: 93.3% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-06-06T08:33:43.742 INFO:teuthology.orchestra.run.smithi086.stderr: 89.1% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mgr.smithi086.mfkgze.log.gz 2024-06-06T08:33:43.897 INFO:teuthology.orchestra.run.smithi086.stderr: 90.5% -- replaced with /var/log/ceph/a8550dee-23dd-11ef-bc9b-c7b262605968/ceph-mon.smithi086.log.gz 2024-06-06T08:33:43.898 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-06-06T08:33:43.899 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m0.289s 2024-06-06T08:33:43.899 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m0.645s 2024-06-06T08:33:43.899 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.040s 2024-06-06T08:33:43.899 INFO:tasks.cephadm:Archiving logs... 2024-06-06T08:33:43.899 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240/remote/smithi086/log 2024-06-06T08:33:43.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-06T08:33:44.071 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240/remote/smithi154/log 2024-06-06T08:33:44.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-06T08:33:44.142 INFO:tasks.cephadm:Removing cluster... 2024-06-06T08:33:44.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 --force 2024-06-06T08:33:44.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8550dee-23dd-11ef-bc9b-c7b262605968 --force 2024-06-06T08:33:45.091 INFO:tasks.cephadm:Removing cephadm ... 2024-06-06T08:33:45.091 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-06T08:33:45.108 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-06T08:33:45.124 INFO:tasks.cephadm:Teardown complete 2024-06-06T08:33:45.124 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-06T08:33:45.306 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1621612819e545a58ca30ace16f3ed30 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-06T08:33:45.308 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-06T08:33:45.320 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-06T08:33:45.419 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-06T08:33:45.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-06T08:33:45.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-06T08:33:45.506 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-06T08:33:45.514 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-06T08:33:45.514 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T08:33:45.517 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T08:33:45.531 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-06-06T08:33:45.532 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-06-06T08:33:45.536 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T08:33:45.536 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-06-06T08:33:45.536 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv01.front.sepia.ceph.com 3 6 377 0 +12us[ +32us] +/- 65ms 2024-06-06T08:33:45.536 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 -1953us[-1943us] +/- 59ms 2024-06-06T08:33:45.537 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 +2376us[+2386us] +/- 70ms 2024-06-06T08:33:45.537 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:33:45.538 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T08:33:45.539 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-06-06T08:33:45.539 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv01.front.sepia.ceph.com 3 6 377 0 +15us[ +31us] +/- 65ms 2024-06-06T08:33:45.539 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 62 -1934us[-1925us] +/- 59ms 2024-06-06T08:33:45.539 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 +2370us[+2379us] +/- 70ms 2024-06-06T08:33:45.539 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T08:33:45.539 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-06T08:33:45.592 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-06T08:33:45.593 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-06T08:33:45.670 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-06T08:33:45.746 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-06T08:33:45.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T08:33:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T08:33:45.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T08:33:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T08:33:45.870 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-06T08:33:45.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-06T08:33:45.919 INFO:teuthology.task.internal:Duration was 1170.205581 seconds 2024-06-06T08:33:45.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-06T08:33:46.007 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-06T08:33:46.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-06T08:33:46.010 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-06T08:33:46.047 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T08:33:46.049 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T08:33:46.461 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-06T08:33:46.461 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-06-06T08:33:46.461 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-06T08:33:46.482 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-06-06T08:33:46.482 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-06T08:33:46.531 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-06T08:33:46.531 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-06T08:33:46.534 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-06T08:33:46.612 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-06T08:33:46.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-06T08:33:46.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-06T08:33:47.001 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-06T08:33:47.012 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-06T08:33:47.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-06T08:33:47.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-06T08:33:47.062 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-06T08:33:47.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-06T08:33:47.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-06T08:33:47.118 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-06-06T08:33:47.129 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-06-06T08:33:47.168 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-06T08:33:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T08:33:47.213 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-06T08:33:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T08:33:47.227 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-06T08:33:47.241 INFO:teuthology.task.internal:Transferring archived files... 2024-06-06T08:33:47.242 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240/remote/smithi086 2024-06-06T08:33:47.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-06T08:33:47.294 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743240/remote/smithi154 2024-06-06T08:33:47.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-06T08:33:47.329 INFO:teuthology.task.internal:Removing archive directory... 2024-06-06T08:33:47.329 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-06T08:33:47.332 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-06T08:33:47.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-06T08:33:47.390 INFO:teuthology.task.internal:Not uploading archives. 2024-06-06T08:33:47.390 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-06T08:33:47.450 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-06T08:33:47.451 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-06T08:33:47.453 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-06T08:33:47.465 INFO:teuthology.orchestra.run.smithi086.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 6 08:33 /home/ubuntu/cephtest 2024-06-06T08:33:47.468 INFO:teuthology.orchestra.run.smithi154.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 6 08:33 /home/ubuntu/cephtest 2024-06-06T08:33:47.469 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-06T08:33:47.503 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-06T08:33:47.756 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1170.2055807113647 failure_reason: '"1717662227.7967122 mgr.smithi154.kxiapq (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1621612819e545a58ca30ace16f3ed30 status: fail success: false 2024-06-06T08:33:47.757 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-06T08:33:47.847 INFO:teuthology.run:FAIL