2024-06-06T07:25:07.251 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-06T07:25:07.252 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --name teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208 --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208/orig.config.yaml 2024-06-06T07:25:07.318 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-06T07:25:07.403 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208 branch: squid description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7743208' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/reef ' name: teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 3d746ca5346f34eef7e826a1eb71e59e02b682ae owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5050 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 416/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 3d746ca5346f34eef7e826a1eb71e59e02b682ae targets: smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG4AiRD9C1emJ9yMjw5z0GtWUOko12v92/8jzjD37b7H5oc46wIL27JCDXMMc71yLHLOMqWW98WEEcQKISJ5gIQ= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCxs+v2jfCbU6DXMlLmerpkLqS9vrjaT221E0RGx3Q43tv7o5lXprW5DQR1b1LyVlutvvVI+uLhLBOf7B+PaDGE= tasks: - install: branch: reef exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef roleless: true - print: '**** done end installing reef cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-05_21:24:16 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-06-06T07:25:07.403 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa; will attempt to use it 2024-06-06T07:25:07.404 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks 2024-06-06T07:25:07.404 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-06T07:25:07.406 INFO:teuthology.task.internal:Checking packages... 2024-06-06T07:25:07.429 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-06T07:25:07.429 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T07:25:07.429 INFO:teuthology.packaging:ref: None 2024-06-06T07:25:07.430 INFO:teuthology.packaging:tag: None 2024-06-06T07:25:07.430 INFO:teuthology.packaging:branch: squid 2024-06-06T07:25:07.430 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T07:25:07.430 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-06T07:25:07.729 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2960.g3d746ca5 2024-06-06T07:25:07.731 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-06T07:25:07.737 INFO:teuthology.task.internal:no buildpackages task found 2024-06-06T07:25:07.737 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-06T07:25:07.770 INFO:teuthology.task.internal:Saving configuration 2024-06-06T07:25:07.791 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-06T07:25:07.797 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-06T07:25:07.823 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-06 07:20:07.698855', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG4AiRD9C1emJ9yMjw5z0GtWUOko12v92/8jzjD37b7H5oc46wIL27JCDXMMc71yLHLOMqWW98WEEcQKISJ5gIQ='} 2024-06-06T07:25:07.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-06 07:20:07.700652', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCxs+v2jfCbU6DXMlLmerpkLqS9vrjaT221E0RGx3Q43tv7o5lXprW5DQR1b1LyVlutvvVI+uLhLBOf7B+PaDGE='} 2024-06-06T07:25:07.849 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-06T07:25:07.855 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-06-06T07:25:07.855 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-06-06T07:25:07.855 INFO:teuthology.run_tasks:Running task console_log... 2024-06-06T07:25:07.938 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4beec4fbe0>, signals=[15]) 2024-06-06T07:25:07.938 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-06T07:25:07.943 INFO:teuthology.task.internal:Opening connections... 2024-06-06T07:25:07.943 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-06-06T07:25:07.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:25:08.023 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-06-06T07:25:08.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:25:08.103 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-06T07:25:08.108 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-06-06T07:25:08.125 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-06-06T07:25:08.126 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-06-06T07:25:08.182 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-06-06T07:25:08.182 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="9" 2024-06-06T07:25:08.182 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-06-06T07:25:08.182 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-06-06T07:25:08.182 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="9" 2024-06-06T07:25:08.182 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el9" 2024-06-06T07:25:08.182 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-06T07:25:08.183 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-06-06T07:25:08.183 INFO:teuthology.orchestra.run.smithi082.stdout:LOGO="fedora-logo-icon" 2024-06-06T07:25:08.183 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-06T07:25:08.183 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-06-06T07:25:08.183 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-06T07:25:08.183 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-06T07:25:08.183 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-06T07:25:08.184 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-06-06T07:25:08.213 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-06-06T07:25:08.231 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-06-06T07:25:08.231 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-06-06T07:25:08.287 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-06T07:25:08.288 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-06T07:25:08.289 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-06-06T07:25:08.313 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-06T07:25:08.321 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-06T07:25:08.385 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-06T07:25:08.385 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-06-06T07:25:08.388 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-06-06T07:25:08.404 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-06T07:25:08.411 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-06T07:25:08.411 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-06-06T07:25:08.444 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-06-06T07:25:08.468 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-06T07:25:08.526 INFO:teuthology.run_tasks:Running task kernel... 2024-06-06T07:25:08.540 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-06T07:25:08.540 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-06T07:25:08.540 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-06T07:25:08.541 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-06T07:25:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T07:25:08.557 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-06-06T07:25:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T07:25:08.558 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-06-06T07:25:08.613 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-437.el9.x86_64 2024-06-06T07:25:08.613 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.14.0-437.el9.x86_64 2024-06-06T07:25:08.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2024-06-06T07:25:08.614 INFO:teuthology.orchestra.run.smithi082.stdout:5.14.0-437.el9.x86_64 2024-06-06T07:25:08.614 INFO:teuthology.task.kernel:Running kernel on smithi082: 5.14.0-437.el9.x86_64 2024-06-06T07:25:08.615 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2024-06-06T07:25:10.821 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.1 MB 00:00 2024-06-06T07:25:10.938 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.1 MB 00:00 2024-06-06T07:25:12.572 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-06-06T07:25:13.039 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-06-06T07:25:16.673 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-06-06T07:25:17.305 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-06-06T07:25:18.348 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 17 kB 00:00 2024-06-06T07:25:18.980 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 17 kB 00:00 2024-06-06T07:25:19.851 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-06-06T07:25:20.216 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-06-06T07:25:25.065 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-06-06T07:25:25.613 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 5.7 kB/s | 1.7 kB 00:00 2024-06-06T07:25:27.121 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-06T07:25:27.121 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-06T07:25:27.220 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-06-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:25:27.227 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2024-06-06T07:25:27.228 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:25:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 80 M 2024-06-06T07:25:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 124 M 2024-06-06T07:25:27.229 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:25:27.581 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:25:27.583 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:25:27.589 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:25:27.652 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-06T07:25:27.652 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-06T07:25:27.736 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:25:27.741 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:25:27.742 INFO:teuthology.orchestra.run.smithi082.stdout:Install 4 Packages 2024-06-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 80 M 2024-06-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 124 M 2024-06-06T07:25:27.743 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:25:27.774 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 3.2 MB/s | 893 kB 00:00 2024-06-06T07:25:27.884 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:25:27.886 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:25:27.893 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:25:28.031 INFO:teuthology.orchestra.run.smithi082.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 3.9 MB/s | 893 kB 00:00 2024-06-06T07:25:28.075 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-06-06T07:25:28.290 INFO:teuthology.orchestra.run.smithi082.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-06-06T07:25:28.517 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 39 MB/s | 29 MB 00:00 2024-06-06T07:25:28.682 INFO:teuthology.orchestra.run.smithi082.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 44 MB/s | 29 MB 00:00 2024-06-06T07:25:28.867 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 25 MB/s | 35 MB 00:01 2024-06-06T07:25:28.869 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:25:28.870 INFO:teuthology.orchestra.run.smithi184.stdout:Total 49 MB/s | 80 MB 00:01 2024-06-06T07:25:28.998 INFO:teuthology.orchestra.run.smithi082.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-06-06T07:25:29.001 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:25:29.001 INFO:teuthology.orchestra.run.smithi082.stdout:Total 64 MB/s | 80 MB 00:01 2024-06-06T07:25:29.563 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:25:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:25:29.895 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:25:30.041 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:25:30.042 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:25:30.838 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:25:30.838 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:25:30.967 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:25:30.968 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:25:33.234 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:25:33.349 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:25:33.845 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-06T07:25:33.926 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T07:25:34.041 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-06T07:25:34.132 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T07:25:36.547 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T07:25:36.706 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T07:25:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T07:25:36.884 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T07:25:41.503 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:25:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T07:25:41.705 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:25:46.266 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:25:46.482 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:26:15.513 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:26:15.552 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:26:15.692 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:26:15.731 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:26:15.845 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:26:15.845 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-06T07:26:15.845 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T07:26:15.845 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T07:26:16.037 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:26:16.037 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-06T07:26:16.037 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T07:26:16.037 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T07:26:16.690 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:26:16.690 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:26:16.691 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:26:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-06T07:26:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-06T07:26:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-06T07:26:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-06T07:26:16.692 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:26:16.692 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:26:16.821 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T07:26:16.821 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:26:16.822 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:26:16.822 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-06T07:26:16.822 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-06T07:26:16.822 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-06T07:26:16.822 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-06T07:26:16.822 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:26:16.823 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:26:17.045 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T07:26:17.130 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T07:26:17.131 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T07:26:17.131 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-06T07:26:17.131 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-06T07:26:17.131 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-06T07:26:17.131 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-06-06T07:26:17.131 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-06-06T07:26:17.132 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T07:26:17.178 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T07:26:17.183 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T07:26:17.184 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T07:26:17.184 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2024-06-06T07:26:17.214 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2024-06-06T07:26:17.214 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2024-06-06T07:26:17.214 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.06 2024-06-06T07:26:17.214 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 68.el9 2024-06-06T07:26:17.214 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2024-06-06T07:26:17.214 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-06T07:26:17.214 INFO:teuthology.orchestra.run.smithi184.stdout:Group : Unspecified 2024-06-06T07:26:17.214 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 8264254 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : builder@centos.org 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-06T07:26:17.215 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-06T07:26:17.216 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-06T07:26:17.216 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2024-06-06T07:26:17.216 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:26:17.216 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2024-06-06T07:26:17.217 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-06T07:26:17.217 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 5.14.0-452.el9.x86_64 2024-06-06T07:26:17.217 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-06T07:26:17.270 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T07:26:17.270 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T07:26:17.271 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-06T07:26:17.271 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-06T07:26:17.271 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-06T07:26:17.271 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-06-06T07:26:17.271 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2024-06-06T07:26:17.271 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T07:26:17.323 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T07:26:17.323 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T07:26:17.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo rpm -qi grub2-tools 2024-06-06T07:26:17.363 INFO:teuthology.orchestra.run.smithi082.stdout:Name : grub2-tools 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Epoch : 1 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Version : 2.06 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Release : 68.el9 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: x86_64 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Group : Unspecified 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Size : 8264254 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:License : GPLv3+ 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-06T07:26:17.364 INFO:teuthology.orchestra.run.smithi082.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:Packager : builder@centos.org 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:Vendor : CentOS 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:Summary : Support tools for GRUB. 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:Description : 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:hardware devices. 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:26:17.365 INFO:teuthology.orchestra.run.smithi082.stdout:This subpackage provides tools for support of all platforms. 2024-06-06T07:26:17.366 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-06T07:26:17.366 INFO:teuthology.task.kernel:Updating grub on smithi082 to boot 5.14.0-452.el9.x86_64 2024-06-06T07:26:17.367 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-06T07:26:17.822 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2024-06-06T07:26:17.988 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2024-06-06T07:26:19.225 INFO:teuthology.orchestra.run.smithi184.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-06T07:26:19.266 INFO:teuthology.orchestra.run.smithi184.stderr:done 2024-06-06T07:26:19.269 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-06-06T07:26:19.285 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.3Mk0Vf6vwp 2024-06-06T07:26:19.285 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3Mk0Vf6vwp 2024-06-06T07:26:19.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.3Mk0Vf6vwp 2024-06-06T07:26:19.373 INFO:teuthology.orchestra.run.smithi082.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-06T07:26:19.399 INFO:teuthology.orchestra.run.smithi082.stderr:done 2024-06-06T07:26:19.401 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-06-06T07:26:19.431 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.XUPxL0NfMv 2024-06-06T07:26:19.431 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XUPxL0NfMv 2024-06-06T07:26:19.486 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.3Mk0Vf6vwp is 6KB 2024-06-06T07:26:19.497 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.XUPxL0NfMv 2024-06-06T07:26:19.536 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.3Mk0Vf6vwp 2024-06-06T07:26:19.550 DEBUG:teuthology.orchestra.run.smithi184:> sudo /bin/ls /boot/loader/entries || true 2024-06-06T07:26:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-06T07:26:19.610 INFO:teuthology.orchestra.run.smithi184.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-06T07:26:19.611 INFO:teuthology.orchestra.run.smithi184.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-06T07:26:19.611 INFO:teuthology.orchestra.run.smithi184.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-06T07:26:19.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-06T07:26:19.630 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.XUPxL0NfMv is 6KB 2024-06-06T07:26:19.679 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.XUPxL0NfMv 2024-06-06T07:26:19.693 DEBUG:teuthology.orchestra.run.smithi082:> sudo /bin/ls /boot/loader/entries || true 2024-06-06T07:26:19.754 INFO:teuthology.orchestra.run.smithi082.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-06T07:26:19.754 INFO:teuthology.orchestra.run.smithi082.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-06T07:26:19.754 INFO:teuthology.orchestra.run.smithi082.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-06T07:26:19.754 INFO:teuthology.orchestra.run.smithi082.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-06T07:26:19.756 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2024-06-06T07:26:19.756 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-06T07:26:19.904 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2024-06-06T07:26:49.786 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-06-06T07:26:49.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-06-06T07:26:49.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:26:49.907 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-06-06T07:26:49.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-06-06T07:26:49.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:27:08.319 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-06-06T07:27:08.319 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-06-06T07:27:17.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-06-06T07:27:17.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:27:17.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-06-06T07:27:17.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:27:20.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-06-06T07:27:20.383 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-06-06T07:27:32.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-06-06T07:27:32.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:27:32.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-06-06T07:27:32.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:27:50.819 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-06-06T07:27:50.819 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-06-06T07:28:05.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-06-06T07:28:05.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:28:05.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-06-06T07:28:05.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:28:06.150 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-06-06T07:28:06.335 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-06-06T07:28:06.552 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-06-06T07:28:06.552 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-452.el9.x86_64"... 2024-06-06T07:28:06.552 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-06-06T07:28:06.570 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-452.el9.x86_64 2024-06-06T07:28:06.570 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-06T07:28:06.570 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-06T07:28:06.571 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-06T07:28:06.675 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-06-06T07:28:06.676 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-452.el9.x86_64"... 2024-06-06T07:28:06.676 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-06-06T07:28:06.693 INFO:teuthology.orchestra.run.smithi082.stdout:5.14.0-452.el9.x86_64 2024-06-06T07:28:06.693 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-06T07:28:06.693 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-06T07:28:06.693 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-06T07:28:07.572 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-06-06T07:28:07.572 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-06T07:28:07.679 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2024-06-06T07:28:07.694 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-06-06T07:28:07.694 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-06T07:28:07.716 DEBUG:teuthology.parallel:result is None 2024-06-06T07:28:07.854 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2024-06-06T07:28:07.910 DEBUG:teuthology.parallel:result is None 2024-06-06T07:28:07.911 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-06T07:28:07.920 INFO:teuthology.task.internal:Creating test directory... 2024-06-06T07:28:07.920 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-06T07:28:07.923 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-06T07:28:07.963 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-06T07:28:07.971 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-06T07:28:08.001 INFO:teuthology.task.internal:Creating archive directory... 2024-06-06T07:28:08.001 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-06T07:28:08.009 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-06T07:28:08.062 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-06T07:28:08.074 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-06T07:28:08.074 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-06T07:28:08.077 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-06T07:28:08.150 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T07:28:08.160 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T07:28:08.219 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T07:28:08.242 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T07:28:08.242 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-06T07:28:08.257 INFO:teuthology.task.internal:Configuring sudo... 2024-06-06T07:28:08.257 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-06T07:28:08.259 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-06T07:28:08.326 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-06T07:28:08.339 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-06T07:28:08.340 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-06T07:28:08.342 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-06T07:28:08.381 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T07:28:08.505 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T07:28:08.581 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T07:28:08.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T07:28:08.723 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:28:08.723 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-06T07:28:08.792 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T07:28:08.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T07:28:08.895 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T07:28:08.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T07:28:09.017 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:28:09.017 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-06T07:28:09.081 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-06-06T07:28:09.083 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-06-06T07:28:09.167 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T07:28:09.279 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T07:28:09.631 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-06T07:28:09.639 INFO:teuthology.task.internal:Starting timer... 2024-06-06T07:28:09.640 INFO:teuthology.run_tasks:Running task pcp... 2024-06-06T07:28:09.691 INFO:teuthology.run_tasks:Running task selinux... 2024-06-06T07:28:09.731 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-06T07:28:09.731 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-06-06T07:28:09.780 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-06-06T07:28:09.782 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2024-06-06T07:28:09.830 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2024-06-06T07:28:09.832 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-06T07:28:09.832 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-06-06T07:28:09.850 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-06-06T07:28:09.850 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2024-06-06T07:28:09.879 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2024-06-06T07:28:09.880 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi082.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi184.front.sepia.ceph.com': 'permissive'} 2024-06-06T07:28:09.880 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T07:28:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T07:28:09.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T07:28:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T07:28:09.947 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-06T07:28:09.947 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-06-06T07:28:09.989 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2024-06-06T07:28:10.016 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-06T07:28:10.026 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-06T07:28:10.039 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-06T07:28:10.040 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi082.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-06T07:32:28.862 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi082.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2024-06-06T07:32:28.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-06-06T07:32:28.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:32:28.948 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-06-06T07:32:29.025 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-06-06T07:32:29.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-06-06T07:32:29.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T07:32:29.105 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-06-06T07:32:29.182 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-06-06T07:32:29.182 INFO:teuthology.run_tasks:Running task clock... 2024-06-06T07:32:29.197 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-06T07:32:29.197 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-06T07:32:29.198 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T07:32:29.200 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-06T07:32:29.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T07:32:29.229 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-06T07:32:29.244 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-06T07:32:29.253 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-06T07:32:29.267 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-06T07:32:29.269 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-06-06T07:32:29.280 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-06-06T07:32:29.292 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2024-06-06T07:32:29.294 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-06T07:32:29.303 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2024-06-06T07:32:29.308 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-06T07:32:29.317 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-06T07:32:29.330 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-06T07:32:29.361 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-06-06T07:32:29.365 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T07:32:29.365 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-06-06T07:32:29.365 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:32:29.365 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:32:29.366 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:32:29.366 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:32:29.384 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-06-06T07:32:29.388 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T07:32:29.388 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-06-06T07:32:29.388 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:32:29.388 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:32:29.388 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:32:29.388 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:32:29.389 INFO:teuthology.run_tasks:Running task install... 2024-06-06T07:32:29.401 DEBUG:teuthology.task.install:project ceph 2024-06-06T07:32:29.401 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-06T07:32:29.401 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-06T07:32:29.401 INFO:teuthology.task.install:Using flavor: default 2024-06-06T07:32:29.407 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-06T07:32:29.407 INFO:teuthology.task.install:extra packages: [] 2024-06-06T07:32:29.407 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-06T07:32:29.407 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T07:32:29.408 INFO:teuthology.packaging:ref: None 2024-06-06T07:32:29.408 INFO:teuthology.packaging:tag: None 2024-06-06T07:32:29.408 INFO:teuthology.packaging:branch: reef 2024-06-06T07:32:29.408 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T07:32:29.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-06-06T07:32:29.410 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-06T07:32:29.411 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T07:32:29.411 INFO:teuthology.packaging:ref: None 2024-06-06T07:32:29.411 INFO:teuthology.packaging:tag: None 2024-06-06T07:32:29.411 INFO:teuthology.packaging:branch: reef 2024-06-06T07:32:29.411 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T07:32:29.411 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-06-06T07:32:29.671 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/flavors/default/ 2024-06-06T07:32:29.671 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1413.g8a28a0ae 2024-06-06T07:32:29.785 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-06T07:32:29.785 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:32:29.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-06T07:32:29.802 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/flavors/default/ 2024-06-06T07:32:29.802 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1413.g8a28a0ae 2024-06-06T07:32:29.811 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-06T07:32:29.811 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T07:32:29.811 INFO:teuthology.packaging:ref: None 2024-06-06T07:32:29.811 INFO:teuthology.packaging:tag: None 2024-06-06T07:32:29.811 INFO:teuthology.packaging:branch: reef 2024-06-06T07:32:29.812 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T07:32:29.812 DEBUG:teuthology.orchestra.run.smithi184:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-06T07:32:29.877 DEBUG:teuthology.orchestra.run.smithi184:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-06T07:32:29.918 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-06T07:32:29.918 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:32:29.918 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-06T07:32:29.949 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-06T07:32:29.949 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T07:32:29.949 INFO:teuthology.packaging:ref: None 2024-06-06T07:32:29.949 INFO:teuthology.packaging:tag: None 2024-06-06T07:32:29.949 INFO:teuthology.packaging:branch: reef 2024-06-06T07:32:29.949 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T07:32:29.950 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-06T07:32:29.957 DEBUG:teuthology.orchestra.run.smithi184:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-06T07:32:30.017 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-06T07:32:30.021 INFO:teuthology.orchestra.run.smithi184.stdout:check_obsoletes = 1 2024-06-06T07:32:30.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum clean all 2024-06-06T07:32:30.093 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-06T07:32:30.159 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-06-06T07:32:30.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-06-06T07:32:30.246 INFO:teuthology.orchestra.run.smithi184.stdout:45 files removed 2024-06-06T07:32:30.273 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw 2024-06-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi082.stdout:45 files removed 2024-06-06T07:32:30.432 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw 2024-06-06T07:32:30.911 INFO:teuthology.orchestra.run.smithi184.stdout:ceph packages for x86_64 214 kB/s | 77 kB 00:00 2024-06-06T07:32:31.039 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 238 kB/s | 77 kB 00:00 2024-06-06T07:32:31.287 INFO:teuthology.orchestra.run.smithi184.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-06-06T07:32:31.476 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-06-06T07:32:31.737 INFO:teuthology.orchestra.run.smithi184.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-06-06T07:32:31.818 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-06-06T07:32:32.347 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-06-06T07:32:32.428 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-06-06T07:32:34.471 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-06-06T07:32:34.615 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-06-06T07:32:38.688 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-06-06T07:32:39.049 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.2 MB 00:00 2024-06-06T07:32:40.196 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 17 kB 00:00 2024-06-06T07:32:40.532 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 17 kB 00:00 2024-06-06T07:32:42.842 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 21 MB 00:02 2024-06-06T07:32:43.161 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 21 MB 00:02 2024-06-06T07:32:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-06-06T07:32:48.332 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-06-06T07:32:50.620 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-06-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 7.7 M 2024-06-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-06-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 3.3 M 2024-06-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 3.0 M 2024-06-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-06-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 5.1 M 2024-06-06T07:32:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 18 M 2024-06-06T07:32:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 25 k 2024-06-06T07:32:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-06-06T07:32:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-06T07:32:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-06T07:32:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-06T07:32:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 709 k 2024-06-06T07:32:50.636 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 474 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 4.5 M 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 45 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 129 k 2024-06-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 161 k 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 320 k 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 300 k 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 99 k 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout:Install 28 Packages 2024-06-06T07:32:50.638 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 2 Packages 2024-06-06T07:32:50.639 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:32:50.639 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 53 M 2024-06-06T07:32:50.639 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:32:50.783 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:32:50.793 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:32:50.793 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-06-06T07:32:50.793 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:32:50.793 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:32:50.793 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 7.7 M 2024-06-06T07:32:50.793 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrading: 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 3.3 M 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 3.0 M 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 5.1 M 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 18 M 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 25 k 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 709 k 2024-06-06T07:32:50.794 INFO:teuthology.orchestra.run.smithi184.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 474 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2 x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 4.5 M 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 45 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 129 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 161 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 320 k 2024-06-06T07:32:50.795 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 300 k 2024-06-06T07:32:50.796 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 99 k 2024-06-06T07:32:50.796 INFO:teuthology.orchestra.run.smithi184.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-06T07:32:50.796 INFO:teuthology.orchestra.run.smithi184.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-06T07:32:50.796 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:32:50.796 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:32:50.796 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:32:50.796 INFO:teuthology.orchestra.run.smithi184.stdout:Install 28 Packages 2024-06-06T07:32:50.796 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrade 2 Packages 2024-06-06T07:32:50.796 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:32:50.797 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 53 M 2024-06-06T07:32:50.797 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:32:51.514 INFO:teuthology.orchestra.run.smithi082.stdout:(1/30): ceph-base-18.2.2-1413.g8a28a0ae.el9.x86 7.4 MB/s | 5.1 MB 00:00 2024-06-06T07:32:51.614 INFO:teuthology.orchestra.run.smithi082.stdout:(2/30): ceph-selinux-18.2.2-1413.g8a28a0ae.el9. 252 kB/s | 25 kB 00:00 2024-06-06T07:32:51.840 INFO:teuthology.orchestra.run.smithi082.stdout:(3/30): ceph-common-18.2.2-1413.g8a28a0ae.el9.x 18 MB/s | 18 MB 00:01 2024-06-06T07:32:51.957 INFO:teuthology.orchestra.run.smithi082.stdout:(4/30): ceph-radosgw-18.2.2-1413.g8a28a0ae.el9. 6.8 MB/s | 7.7 MB 00:01 2024-06-06T07:32:51.998 INFO:teuthology.orchestra.run.smithi082.stdout:(5/30): libcephfs2-18.2.2-1413.g8a28a0ae.el9.x8 1.8 MB/s | 709 kB 00:00 2024-06-06T07:32:52.040 INFO:teuthology.orchestra.run.smithi082.stdout:(6/30): libradosstriper1-18.2.2-1413.g8a28a0ae. 2.3 MB/s | 474 kB 00:00 2024-06-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi082.stdout:(7/30): python3-ceph-argparse-18.2.2-1413.g8a28 673 kB/s | 45 kB 00:00 2024-06-06T07:32:52.091 INFO:teuthology.orchestra.run.smithi082.stdout:(8/30): python3-ceph-common-18.2.2-1413.g8a28a0 2.5 MB/s | 129 kB 00:00 2024-06-06T07:32:52.116 INFO:teuthology.orchestra.run.smithi082.stdout:(9/30): python3-cephfs-18.2.2-1413.g8a28a0ae.el 3.1 MB/s | 161 kB 00:00 2024-06-06T07:32:52.150 INFO:teuthology.orchestra.run.smithi082.stdout:(10/30): python3-rados-18.2.2-1413.g8a28a0ae.el 5.3 MB/s | 320 kB 00:00 2024-06-06T07:32:52.175 INFO:teuthology.orchestra.run.smithi082.stdout:(11/30): python3-rbd-18.2.2-1413.g8a28a0ae.el9. 5.0 MB/s | 300 kB 00:00 2024-06-06T07:32:52.200 INFO:teuthology.orchestra.run.smithi082.stdout:(12/30): python3-rgw-18.2.2-1413.g8a28a0ae.el9. 1.9 MB/s | 99 kB 00:00 2024-06-06T07:32:52.256 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:32:52.262 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:32:52.343 INFO:teuthology.orchestra.run.smithi082.stdout:(13/30): librgw2-18.2.2-1413.g8a28a0ae.el9.x86_ 12 MB/s | 4.5 MB 00:00 2024-06-06T07:32:52.344 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:32:52.434 INFO:teuthology.orchestra.run.smithi082.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 771 kB/s | 160 kB 00:00 2024-06-06T07:32:52.460 INFO:teuthology.orchestra.run.smithi082.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-06-06T07:32:52.485 INFO:teuthology.orchestra.run.smithi082.stdout:(16/30): boost-program-options-1.75.0-8.el9.x86 345 kB/s | 107 kB 00:00 2024-06-06T07:32:52.561 INFO:teuthology.orchestra.run.smithi082.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 561 kB/s | 42 kB 00:00 2024-06-06T07:32:52.586 INFO:teuthology.orchestra.run.smithi082.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-06-06T07:32:52.611 INFO:teuthology.orchestra.run.smithi082.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 2.2 MB/s | 111 kB 00:00 2024-06-06T07:32:52.636 INFO:teuthology.orchestra.run.smithi082.stdout:(20/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-06-06T07:32:52.686 INFO:teuthology.orchestra.run.smithi082.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-06-06T07:32:52.720 INFO:teuthology.orchestra.run.smithi082.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-06-06T07:32:52.745 INFO:teuthology.orchestra.run.smithi082.stdout:(23/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 238 kB/s | 26 kB 00:00 2024-06-06T07:32:52.771 INFO:teuthology.orchestra.run.smithi082.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-06T07:32:52.804 INFO:teuthology.orchestra.run.smithi082.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 5.7 MB/s | 192 kB 00:00 2024-06-06T07:32:52.871 INFO:teuthology.orchestra.run.smithi082.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-06-06T07:32:52.917 INFO:teuthology.orchestra.run.smithi184.stdout:(1/30): ceph-base-18.2.2-1413.g8a28a0ae.el9.x86 2.7 MB/s | 5.1 MB 00:01 2024-06-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi082.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-06-06T07:32:52.992 INFO:teuthology.orchestra.run.smithi184.stdout:(2/30): ceph-selinux-18.2.2-1413.g8a28a0ae.el9. 336 kB/s | 25 kB 00:00 2024-06-06T07:32:52.997 INFO:teuthology.orchestra.run.smithi082.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-06-06T07:32:53.086 INFO:teuthology.orchestra.run.smithi184.stdout:(3/30): ceph-radosgw-18.2.2-1413.g8a28a0ae.el9. 3.7 MB/s | 7.7 MB 00:02 2024-06-06T07:32:53.147 INFO:teuthology.orchestra.run.smithi082.stdout:(29/30): librados2-18.2.2-1413.g8a28a0ae.el9.x8 12 MB/s | 3.3 MB 00:00 2024-06-06T07:32:53.197 INFO:teuthology.orchestra.run.smithi082.stdout:(30/30): librbd1-18.2.2-1413.g8a28a0ae.el9.x86_ 12 MB/s | 3.0 MB 00:00 2024-06-06T07:32:53.202 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:32:53.202 INFO:teuthology.orchestra.run.smithi082.stdout:Total 21 MB/s | 53 MB 00:02 2024-06-06T07:32:53.218 INFO:teuthology.orchestra.run.smithi184.stdout:(4/30): libcephfs2-18.2.2-1413.g8a28a0ae.el9.x8 3.1 MB/s | 709 kB 00:00 2024-06-06T07:32:53.243 INFO:teuthology.orchestra.run.smithi184.stdout:(5/30): libradosstriper1-18.2.2-1413.g8a28a0ae. 3.0 MB/s | 474 kB 00:00 2024-06-06T07:32:53.268 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:32:53.293 INFO:teuthology.orchestra.run.smithi184.stdout:(6/30): python3-ceph-argparse-18.2.2-1413.g8a28 902 kB/s | 45 kB 00:00 2024-06-06T07:32:53.303 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:32:53.303 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:32:53.360 INFO:teuthology.orchestra.run.smithi184.stdout:(7/30): python3-ceph-common-18.2.2-1413.g8a28a0 1.9 MB/s | 129 kB 00:00 2024-06-06T07:32:53.427 INFO:teuthology.orchestra.run.smithi184.stdout:(8/30): python3-cephfs-18.2.2-1413.g8a28a0ae.el 2.4 MB/s | 161 kB 00:00 2024-06-06T07:32:53.502 INFO:teuthology.orchestra.run.smithi184.stdout:(9/30): python3-rados-18.2.2-1413.g8a28a0ae.el9 4.2 MB/s | 320 kB 00:00 2024-06-06T07:32:53.578 INFO:teuthology.orchestra.run.smithi184.stdout:(10/30): python3-rbd-18.2.2-1413.g8a28a0ae.el9. 3.9 MB/s | 300 kB 00:00 2024-06-06T07:32:53.787 INFO:teuthology.orchestra.run.smithi184.stdout:(11/30): ceph-common-18.2.2-1413.g8a28a0ae.el9. 6.6 MB/s | 18 MB 00:02 2024-06-06T07:32:53.812 INFO:teuthology.orchestra.run.smithi184.stdout:(12/30): python3-rgw-18.2.2-1413.g8a28a0ae.el9. 425 kB/s | 99 kB 00:00 2024-06-06T07:32:53.837 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:32:53.838 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:32:53.866 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:32:53.937 INFO:teuthology.orchestra.run.smithi184.stdout:(13/30): librgw2-18.2.2-1413.g8a28a0ae.el9.x86_ 6.2 MB/s | 4.5 MB 00:00 2024-06-06T07:32:53.942 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:32:53.942 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:32:54.046 INFO:teuthology.orchestra.run.smithi184.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 413 kB/s | 107 kB 00:00 2024-06-06T07:32:54.071 INFO:teuthology.orchestra.run.smithi184.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 344 kB/s | 45 kB 00:00 2024-06-06T07:32:54.105 INFO:teuthology.orchestra.run.smithi184.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 548 kB/s | 160 kB 00:00 2024-06-06T07:32:54.155 INFO:teuthology.orchestra.run.smithi184.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-06-06T07:32:54.180 INFO:teuthology.orchestra.run.smithi184.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-06-06T07:32:54.205 INFO:teuthology.orchestra.run.smithi184.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 2.2 MB/s | 111 kB 00:00 2024-06-06T07:32:54.239 INFO:teuthology.orchestra.run.smithi184.stdout:(20/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-06-06T07:32:54.298 INFO:teuthology.orchestra.run.smithi184.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-06-06T07:32:54.323 INFO:teuthology.orchestra.run.smithi184.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-06-06T07:32:54.348 INFO:teuthology.orchestra.run.smithi184.stdout:(23/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-06-06T07:32:54.374 INFO:teuthology.orchestra.run.smithi184.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-06T07:32:54.407 INFO:teuthology.orchestra.run.smithi184.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-06-06T07:32:54.466 INFO:teuthology.orchestra.run.smithi184.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-06-06T07:32:54.533 INFO:teuthology.orchestra.run.smithi184.stdout:(27/30): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-06-06T07:32:54.575 INFO:teuthology.orchestra.run.smithi184.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-06-06T07:32:54.892 INFO:teuthology.orchestra.run.smithi184.stdout:(29/30): librbd1-18.2.2-1413.g8a28a0ae.el9.x86_ 8.4 MB/s | 3.0 MB 00:00 2024-06-06T07:32:54.942 INFO:teuthology.orchestra.run.smithi184.stdout:(30/30): librados2-18.2.2-1413.g8a28a0ae.el9.x8 6.9 MB/s | 3.3 MB 00:00 2024-06-06T07:32:54.946 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:32:54.946 INFO:teuthology.orchestra.run.smithi184.stdout:Total 13 MB/s | 53 MB 00:04 2024-06-06T07:32:55.015 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:32:55.049 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:32:55.049 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:32:55.169 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:32:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-06-06T07:32:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-06-06T07:32:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-06-06T07:32:55.585 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:32:55.585 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:32:55.704 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-06-06T07:32:55.751 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 5/32 2024-06-06T07:32:55.843 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 5/32 2024-06-06T07:32:55.927 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:18.2.2-1413.g8a28a0ae.el9.x86_64 6/32 2024-06-06T07:32:55.953 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 7/32 2024-06-06T07:32:56.007 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 7/32 2024-06-06T07:32:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-06-06T07:32:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:18.2.2-1413.g8a28a0ae.el9. 9/32 2024-06-06T07:32:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:18.2.2-1413.g8a28a0ae.el9.x86_64 10/32 2024-06-06T07:32:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:18.2.2-1413.g8a28a0ae.el9.x86_6 11/32 2024-06-06T07:32:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1413.g8a28a0ae.el9.x86_6 11/32 2024-06-06T07:32:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-06-06T07:32:56.379 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-06-06T07:32:56.420 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-06-06T07:32:56.460 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-06-06T07:32:56.487 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-06-06T07:32:56.515 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-06-06T07:32:56.556 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-06-06T07:32:56.857 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-06-06T07:32:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 20/32 2024-06-06T07:32:56.909 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:32:56.979 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-06-06T07:32:56.993 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 20/32 2024-06-06T07:32:57.014 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-06-06T07:32:57.024 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 21/32 2024-06-06T07:32:57.060 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-06-06T07:32:57.071 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-06-06T07:32:57.437 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-06-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-06-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 5/32 2024-06-06T07:32:57.505 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 24/32 2024-06-06T07:32:57.576 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 24/32 2024-06-06T07:32:57.579 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 5/32 2024-06-06T07:32:57.645 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 25/32 2024-06-06T07:32:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-2:18.2.2-1413.g8a28a0ae.el9.x86_64 6/32 2024-06-06T07:32:57.675 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x8 26/32 2024-06-06T07:32:57.706 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 7/32 2024-06-06T07:32:57.760 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 7/32 2024-06-06T07:32:57.798 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-06-06T07:32:57.841 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-argparse-2:18.2.2-1413.g8a28a0ae.el9. 9/32 2024-06-06T07:32:57.907 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-2:18.2.2-1413.g8a28a0ae.el9.x86_64 10/32 2024-06-06T07:32:57.934 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libradosstriper1-2:18.2.2-1413.g8a28a0ae.el9.x86_6 11/32 2024-06-06T07:32:57.983 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1413.g8a28a0ae.el9.x86_6 11/32 2024-06-06T07:32:58.091 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-06-06T07:32:58.134 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-06-06T07:32:58.174 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-06-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-06-06T07:32:58.233 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-06-06T07:32:58.269 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-06-06T07:32:58.302 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-06-06T07:32:58.598 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-06-06T07:32:58.648 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 20/32 2024-06-06T07:32:58.770 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 20/32 2024-06-06T07:32:58.811 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 21/32 2024-06-06T07:32:58.856 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-06-06T07:32:59.257 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-06-06T07:32:59.285 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librgw2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 24/32 2024-06-06T07:32:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librgw2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 24/32 2024-06-06T07:32:59.424 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 25/32 2024-06-06T07:32:59.454 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x8 26/32 2024-06-06T07:32:59.775 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 27/32 2024-06-06T07:32:59.805 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 27/32 2024-06-06T07:33:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 27/32 2024-06-06T07:33:00.384 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 28/32 2024-06-06T07:33:00.471 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 28/32 2024-06-06T07:33:00.471 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-06T07:33:00.471 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-06T07:33:00.471 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/32 2024-06-06T07:33:01.616 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 27/32 2024-06-06T07:33:01.659 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 27/32 2024-06-06T07:33:02.200 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 27/32 2024-06-06T07:33:02.238 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 28/32 2024-06-06T07:33:02.338 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 28/32 2024-06-06T07:33:02.338 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-06T07:33:02.338 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-06T07:33:02.338 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:02.373 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-selinux-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/32 2024-06-06T07:33:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/32 2024-06-06T07:33:10.217 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-06-06T07:33:10.217 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-06-06T07:33:10.217 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-06-06T07:33:10.217 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-06-06T07:33:10.217 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-06-06T07:33:10.217 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /root 2024-06-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-06-06T07:33:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:10.977 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 30/32 2024-06-06T07:33:11.002 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 30/32 2024-06-06T07:33:11.003 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:11.003 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-06T07:33:11.003 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T07:33:11.003 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T07:33:11.003 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:11.004 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-06T07:33:11.088 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-06T07:33:11.088 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-06T07:33:12.055 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/32 2024-06-06T07:33:12.055 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /sys 2024-06-06T07:33:12.055 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /proc 2024-06-06T07:33:12.056 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /mnt 2024-06-06T07:33:12.056 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /var/tmp 2024-06-06T07:33:12.056 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /home 2024-06-06T07:33:12.056 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /root 2024-06-06T07:33:12.056 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /tmp 2024-06-06T07:33:12.056 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:12.155 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-06T07:33:12.155 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/32 2024-06-06T07:33:12.155 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/32 2024-06-06T07:33:12.155 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 3/32 2024-06-06T07:33:12.155 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:18.2.2-1413.g8a28a0ae.el9.x86_64 4/32 2024-06-06T07:33:12.155 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 5/32 2024-06-06T07:33:12.155 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:18.2.2-1413.g8a28a0ae.el9.x86_6 6/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 7/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1413.g8a28a0ae.el9. 8/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x8 9/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:18.2.2-1413.g8a28a0ae.el9.x86_64 10/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:18.2.2-1413.g8a28a0ae.el9.x86_64 11/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 12/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 13/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-06-06T07:33:12.156 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-06-06T07:33:12.157 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-06-06T07:33:12.157 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-06-06T07:33:12.157 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-06-06T07:33:12.157 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-06-06T07:33:12.157 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-06-06T07:33:12.157 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-06-06T07:33:12.157 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-06-06T07:33:12.157 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-06-06T07:33:12.157 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/32 2024-06-06T07:33:12.158 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-06-06T07:33:12.158 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 31/32 2024-06-06T07:33:12.824 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 30/32 2024-06-06T07:33:12.850 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 30/32 2024-06-06T07:33:12.850 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:12.850 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-06T07:33:12.850 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T07:33:12.850 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T07:33:12.851 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:12.852 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-06T07:33:12.936 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-06T07:33:12.936 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-06T07:33:12.994 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-06-06T07:33:12.994 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:12.995 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-06-06T07:33:12.995 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.995 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.995 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:33:12.995 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-06T07:33:12.995 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.995 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.996 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.996 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.996 INFO:teuthology.orchestra.run.smithi082.stdout: fmt-8.1.1-5.el9.x86_64 2024-06-06T07:33:12.996 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-06T07:33:12.996 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-06T07:33:12.996 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-06T07:33:12.996 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.997 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-06T07:33:12.997 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-06T07:33:12.997 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-06T07:33:12.997 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.997 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-06T07:33:12.997 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.998 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-06T07:33:12.998 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-06T07:33:12.998 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-06T07:33:12.998 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.998 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.998 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.998 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-06T07:33:12.999 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.999 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.999 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:12.999 INFO:teuthology.orchestra.run.smithi082.stdout: re2-1:20211101-3.el9.x86_64 2024-06-06T07:33:12.999 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-06T07:33:12.999 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:12.999 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:33:13.324 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-test 2024-06-06T07:33:13.904 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:25 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:14.024 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:33:14.025 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:14.026 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-06-06T07:33:14.026 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:14.026 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:33:14.026 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 40 M 2024-06-06T07:33:14.026 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-06-06T07:33:14.027 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 166 k 2024-06-06T07:33:14.027 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-06T07:33:14.027 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-06T07:33:14.027 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:14.027 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:33:14.027 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:14.028 INFO:teuthology.orchestra.run.smithi082.stdout:Install 4 Packages 2024-06-06T07:33:14.028 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:14.028 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 40 M 2024-06-06T07:33:14.028 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 163 M 2024-06-06T07:33:14.028 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:33:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-06T07:33:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/32 2024-06-06T07:33:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/32 2024-06-06T07:33:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 3/32 2024-06-06T07:33:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-selinux-2:18.2.2-1413.g8a28a0ae.el9.x86_64 4/32 2024-06-06T07:33:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 5/32 2024-06-06T07:33:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libradosstriper1-2:18.2.2-1413.g8a28a0ae.el9.x86_6 6/32 2024-06-06T07:33:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librgw2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 7/32 2024-06-06T07:33:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1413.g8a28a0ae.el9. 8/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x8 9/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-2:18.2.2-1413.g8a28a0ae.el9.x86_64 10/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-2:18.2.2-1413.g8a28a0ae.el9.x86_64 11/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 12/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 13/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-06-06T07:33:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-06-06T07:33:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 31/32 2024-06-06T07:33:14.378 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] socat-1.7.4.1-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:14.685 INFO:teuthology.orchestra.run.smithi082.stdout:(1/4): libcephsqlite-18.2.2-1413.g8a28a0ae.el9. 432 kB/s | 166 kB 00:00 2024-06-06T07:33:14.710 INFO:teuthology.orchestra.run.smithi082.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 744 kB/s | 305 kB 00:00 2024-06-06T07:33:14.710 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:14.769 INFO:teuthology.orchestra.run.smithi082.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 764 kB/s | 64 kB 00:00 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout:Upgraded: 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.952 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: fmt-8.1.1-5.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-06T07:33:14.953 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: re2-1:20211101-3.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:14.954 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:33:15.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-test 2024-06-06T07:33:15.906 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:27 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:16.026 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 40 M 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 166 k 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:16.027 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:33:16.028 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:16.028 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2024-06-06T07:33:16.028 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:16.028 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 40 M 2024-06-06T07:33:16.028 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 163 M 2024-06-06T07:33:16.028 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:33:16.078 INFO:teuthology.orchestra.run.smithi082.stdout:(4/4): ceph-test-18.2.2-1413.g8a28a0ae.el9.x86_ 22 MB/s | 40 MB 00:01 2024-06-06T07:33:16.081 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:16.081 INFO:teuthology.orchestra.run.smithi082.stdout:Total 20 MB/s | 40 MB 00:02 2024-06-06T07:33:16.108 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:33:16.128 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:33:16.128 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:33:16.382 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] socat-1.7.4.1-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:16.444 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:33:16.445 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:33:16.526 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-06-06T07:33:16.528 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:16.568 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-06T07:33:16.668 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): libcephsqlite-18.2.2-1413.g8a28a0ae.el9. 453 kB/s | 166 kB 00:00 2024-06-06T07:33:16.923 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:33:17.007 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-06T07:33:17.052 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-06T07:33:17.079 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:18.2.2-1413.g8a28a0ae.el9.x86_64 3/4 2024-06-06T07:33:17.618 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): ceph-test-18.2.2-1413.g8a28a0ae.el9.x86_ 30 MB/s | 40 MB 00:01 2024-06-06T07:33:17.622 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:17.622 INFO:teuthology.orchestra.run.smithi184.stdout:Total 25 MB/s | 40 MB 00:01 2024-06-06T07:33:17.650 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:33:17.669 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:33:17.670 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:33:17.986 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:33:17.986 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:33:18.456 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:33:18.536 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-06T07:33:18.577 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-06T07:33:18.595 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephsqlite-2:18.2.2-1413.g8a28a0ae.el9.x86_64 3/4 2024-06-06T07:33:20.754 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1413.g8a28a0ae.el9.x86_64 3/4 2024-06-06T07:33:20.827 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:18.2.2-1413.g8a28a0ae.el9.x86_64 4/4 2024-06-06T07:33:22.277 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1413.g8a28a0ae.el9.x86_64 3/4 2024-06-06T07:33:22.370 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-2:18.2.2-1413.g8a28a0ae.el9.x86_64 4/4 2024-06-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-test-2:18.2.2-1413.g8a28a0ae.el9.x86_64 4/4 2024-06-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/4 2024-06-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/4 2024-06-06T07:33:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-06T07:33:23.158 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-06T07:33:23.158 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:23.158 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:33:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-06T07:33:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-06T07:33:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:23.159 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:33:23.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph 2024-06-06T07:33:24.033 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:36 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:24.159 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:33:24.190 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 6.5 k 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 2.1 M 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 1.5 M 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 246 k 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 4.7 M 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 17 M 2024-06-06T07:33:24.191 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-06T07:33:24.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-06T07:33:24.193 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-06T07:33:24.194 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-06T07:33:24.194 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:24.194 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:33:24.194 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:24.194 INFO:teuthology.orchestra.run.smithi082.stdout:Install 33 Packages 2024-06-06T07:33:24.194 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:24.194 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 29 M 2024-06-06T07:33:24.194 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 104 M 2024-06-06T07:33:24.195 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:33:24.313 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-test-2:18.2.2-1413.g8a28a0ae.el9.x86_64 4/4 2024-06-06T07:33:24.313 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/4 2024-06-06T07:33:24.313 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephsqlite-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/4 2024-06-06T07:33:24.313 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-06T07:33:24.590 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-06T07:33:24.591 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:24.591 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:33:24.591 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:24.591 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:24.591 INFO:teuthology.orchestra.run.smithi184.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-06T07:33:24.591 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-06T07:33:24.591 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:24.591 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:33:24.891 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph 2024-06-06T07:33:24.911 INFO:teuthology.orchestra.run.smithi082.stdout:(1/33): ceph-18.2.2-1413.g8a28a0ae.el9.x86_64.r 21 kB/s | 6.5 kB 00:00 2024-06-06T07:33:24.978 INFO:teuthology.orchestra.run.smithi082.stdout:(2/33): ceph-mds-18.2.2-1413.g8a28a0ae.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-06-06T07:33:25.095 INFO:teuthology.orchestra.run.smithi082.stdout:(3/33): ceph-mgr-18.2.2-1413.g8a28a0ae.el9.x86_ 3.0 MB/s | 1.5 MB 00:00 2024-06-06T07:33:25.467 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:37 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:25.593 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:33:25.595 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:25.595 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-06-06T07:33:25.595 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:25.595 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:33:25.595 INFO:teuthology.orchestra.run.smithi184.stdout: ceph x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 6.5 k 2024-06-06T07:33:25.595 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 2.1 M 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 1.5 M 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 246 k 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 4.7 M 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 17 M 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-06T07:33:25.596 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections 2024-06-06T07:33:25.597 INFO:teuthology.orchestra.run.smithi184.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-06T07:33:25.597 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-06T07:33:25.597 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-06T07:33:25.597 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-06T07:33:25.597 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-06T07:33:25.597 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-06T07:33:25.597 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-06T07:33:25.598 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-06T07:33:25.599 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:25.599 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:33:25.599 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:25.599 INFO:teuthology.orchestra.run.smithi184.stdout:Install 33 Packages 2024-06-06T07:33:25.599 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:25.599 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 29 M 2024-06-06T07:33:25.600 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 104 M 2024-06-06T07:33:25.600 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:33:25.612 INFO:teuthology.orchestra.run.smithi082.stdout:(4/33): ceph-osd-18.2.2-1413.g8a28a0ae.el9.x86_ 26 MB/s | 17 MB 00:00 2024-06-06T07:33:25.637 INFO:teuthology.orchestra.run.smithi082.stdout:(5/33): ceph-mgr-modules-core-18.2.2-1413.g8a28 454 kB/s | 246 kB 00:00 2024-06-06T07:33:25.691 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:25.700 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:25.796 INFO:teuthology.orchestra.run.smithi082.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-06-06T07:33:25.796 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:25.837 INFO:teuthology.orchestra.run.smithi082.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-06-06T07:33:25.871 INFO:teuthology.orchestra.run.smithi082.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 5.8 MB/s | 1.2 MB 00:00 2024-06-06T07:33:25.896 INFO:teuthology.orchestra.run.smithi082.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-06-06T07:33:25.922 INFO:teuthology.orchestra.run.smithi082.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-06-06T07:33:25.947 INFO:teuthology.orchestra.run.smithi082.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-06-06T07:33:25.947 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:25.950 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:25.983 INFO:teuthology.orchestra.run.smithi082.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 172 kB 00:00 2024-06-06T07:33:26.000 INFO:teuthology.orchestra.run.smithi082.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 661 kB/s | 35 kB 00:00 2024-06-06T07:33:26.000 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-pytz-2021.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pytz-2021.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:26.003 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-toml-0.10.2-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-toml-0.10.2-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:26.033 INFO:teuthology.orchestra.run.smithi082.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-06-06T07:33:26.050 INFO:teuthology.orchestra.run.smithi082.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-06-06T07:33:26.067 INFO:teuthology.orchestra.run.smithi082.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-06-06T07:33:26.142 INFO:teuthology.orchestra.run.smithi082.stdout:(17/33): ceph-mon-18.2.2-1413.g8a28a0ae.el9.x86 3.8 MB/s | 4.7 MB 00:01 2024-06-06T07:33:26.168 INFO:teuthology.orchestra.run.smithi082.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-06-06T07:33:26.210 INFO:teuthology.orchestra.run.smithi082.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 991 kB/s | 18 kB 00:00 2024-06-06T07:33:26.235 INFO:teuthology.orchestra.run.smithi082.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-06-06T07:33:26.252 INFO:teuthology.orchestra.run.smithi082.stdout:(21/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-06-06T07:33:26.269 INFO:teuthology.orchestra.run.smithi082.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 411 kB/s | 23 kB 00:00 2024-06-06T07:33:26.285 INFO:teuthology.orchestra.run.smithi082.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-06-06T07:33:26.303 INFO:teuthology.orchestra.run.smithi082.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 388 kB/s | 20 kB 00:00 2024-06-06T07:33:26.322 INFO:teuthology.orchestra.run.smithi184.stdout:(1/33): ceph-18.2.2-1413.g8a28a0ae.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-06-06T07:33:26.328 INFO:teuthology.orchestra.run.smithi082.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 786 kB/s | 46 kB 00:00 2024-06-06T07:33:26.353 INFO:teuthology.orchestra.run.smithi082.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-06-06T07:33:26.378 INFO:teuthology.orchestra.run.smithi082.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-06-06T07:33:26.404 INFO:teuthology.orchestra.run.smithi082.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-06T07:33:26.429 INFO:teuthology.orchestra.run.smithi082.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-06-06T07:33:26.439 INFO:teuthology.orchestra.run.smithi184.stdout:(2/33): ceph-mds-18.2.2-1413.g8a28a0ae.el9.x86_ 6.0 MB/s | 2.1 MB 00:00 2024-06-06T07:33:26.454 INFO:teuthology.orchestra.run.smithi082.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-06T07:33:26.488 INFO:teuthology.orchestra.run.smithi082.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-06-06T07:33:26.513 INFO:teuthology.orchestra.run.smithi082.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-06-06T07:33:26.546 INFO:teuthology.orchestra.run.smithi082.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-06-06T07:33:26.551 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:26.551 INFO:teuthology.orchestra.run.smithi082.stdout:Total 12 MB/s | 29 MB 00:02 2024-06-06T07:33:26.606 INFO:teuthology.orchestra.run.smithi184.stdout:(3/33): ceph-mgr-18.2.2-1413.g8a28a0ae.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2024-06-06T07:33:26.662 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:33:26.681 INFO:teuthology.orchestra.run.smithi184.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1413.g8a28 3.2 MB/s | 246 kB 00:00 2024-06-06T07:33:26.698 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:33:26.698 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:33:26.723 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:26.948 INFO:teuthology.orchestra.run.smithi184.stdout:(5/33): ceph-osd-18.2.2-1413.g8a28a0ae.el9.x86_ 32 MB/s | 17 MB 00:00 2024-06-06T07:33:26.950 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:27.023 INFO:teuthology.orchestra.run.smithi184.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 741 kB/s | 253 kB 00:00 2024-06-06T07:33:27.025 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:33:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:33:27.074 INFO:teuthology.orchestra.run.smithi184.stdout:(7/33): python3-cryptography-36.0.1-4.el9.x86_6 9.9 MB/s | 1.2 MB 00:00 2024-06-06T07:33:27.099 INFO:teuthology.orchestra.run.smithi184.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-06-06T07:33:27.124 INFO:teuthology.orchestra.run.smithi184.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.7 MB/s | 135 kB 00:00 2024-06-06T07:33:27.150 INFO:teuthology.orchestra.run.smithi184.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-06-06T07:33:27.183 INFO:teuthology.orchestra.run.smithi184.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-06-06T07:33:27.184 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:27.184 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:27.267 INFO:teuthology.orchestra.run.smithi184.stdout:(12/33): ceph-mon-18.2.2-1413.g8a28a0ae.el9.x86 4.9 MB/s | 4.7 MB 00:00 2024-06-06T07:33:27.269 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-pytz-2021.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pytz-2021.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:27.334 INFO:teuthology.orchestra.run.smithi184.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 231 kB/s | 35 kB 00:00 2024-06-06T07:33:27.368 INFO:teuthology.orchestra.run.smithi184.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 509 kB/s | 51 kB 00:00 2024-06-06T07:33:27.393 INFO:teuthology.orchestra.run.smithi184.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 712 kB/s | 42 kB 00:00 2024-06-06T07:33:27.410 INFO:teuthology.orchestra.run.smithi184.stdout:(16/33): python3-mako-1.1.4-6.el9.noarch.rpm 664 kB/s | 172 kB 00:00 2024-06-06T07:33:27.460 INFO:teuthology.orchestra.run.smithi184.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 471 kB/s | 43 kB 00:00 2024-06-06T07:33:27.510 INFO:teuthology.orchestra.run.smithi184.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 482 kB/s | 11 kB 00:00 2024-06-06T07:33:27.535 INFO:teuthology.orchestra.run.smithi184.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-06-06T07:33:27.552 INFO:teuthology.orchestra.run.smithi184.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 446 kB/s | 18 kB 00:00 2024-06-06T07:33:27.570 INFO:teuthology.orchestra.run.smithi184.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 690 kB/s | 23 kB 00:00 2024-06-06T07:33:27.586 INFO:teuthology.orchestra.run.smithi184.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 578 kB/s | 19 kB 00:00 2024-06-06T07:33:27.603 INFO:teuthology.orchestra.run.smithi184.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 589 kB/s | 20 kB 00:00 2024-06-06T07:33:27.620 INFO:teuthology.orchestra.run.smithi184.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-06-06T07:33:27.645 INFO:teuthology.orchestra.run.smithi184.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-06-06T07:33:27.671 INFO:teuthology.orchestra.run.smithi184.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-06-06T07:33:27.696 INFO:teuthology.orchestra.run.smithi184.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-06-06T07:33:27.713 INFO:teuthology.orchestra.run.smithi184.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-06-06T07:33:27.738 INFO:teuthology.orchestra.run.smithi184.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 852 kB/s | 36 kB 00:00 2024-06-06T07:33:27.772 INFO:teuthology.orchestra.run.smithi184.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-06T07:33:27.797 INFO:teuthology.orchestra.run.smithi184.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-06-06T07:33:27.822 INFO:teuthology.orchestra.run.smithi184.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-06-06T07:33:27.864 INFO:teuthology.orchestra.run.smithi184.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-06-06T07:33:27.869 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:27.869 INFO:teuthology.orchestra.run.smithi184.stdout:Total 13 MB/s | 29 MB 00:02 2024-06-06T07:33:27.989 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:33:28.026 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:33:28.026 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:33:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:33:28.324 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-06T07:33:28.356 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-06T07:33:28.389 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-06T07:33:28.400 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:33:28.401 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:33:28.482 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-06T07:33:28.568 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-06T07:33:28.617 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-06T07:33:28.660 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-06T07:33:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-06T07:33:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-06T07:33:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-06T07:33:28.802 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-06T07:33:28.842 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-06T07:33:28.889 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-06T07:33:28.922 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-06T07:33:28.996 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-06T07:33:29.142 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-06T07:33:29.223 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-06T07:33:29.301 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-06T07:33:29.347 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-06T07:33:29.405 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-06T07:33:29.459 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-06T07:33:29.564 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:33:29.626 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-06T07:33:29.648 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-06T07:33:29.666 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-06T07:33:29.699 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-06T07:33:29.701 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-06T07:33:29.778 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-06T07:33:29.791 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-06T07:33:29.866 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-06T07:33:29.878 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-06T07:33:29.900 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-06T07:33:29.918 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-06T07:33:29.970 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-06T07:33:30.000 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-06T07:33:30.015 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-06T07:33:30.035 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-06T07:33:30.093 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-06T07:33:30.134 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-06T07:33:30.169 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-06T07:33:30.174 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1413.g8a28a0ae.el9. 28/33 2024-06-06T07:33:30.202 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/33 2024-06-06T07:33:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-06T07:33:30.227 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/33 2024-06-06T07:33:30.227 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:30.227 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-06T07:33:30.227 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T07:33:30.228 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T07:33:30.228 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:30.241 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-06T07:33:30.314 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-06T07:33:30.461 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-06T07:33:30.541 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-06T07:33:30.594 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-06T07:33:30.633 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-06T07:33:30.690 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-06T07:33:30.745 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-06T07:33:30.941 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-06T07:33:31.004 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-06T07:33:31.090 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-06T07:33:31.179 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-06T07:33:31.219 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-06T07:33:31.336 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-06T07:33:31.487 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1413.g8a28a0ae.el9. 28/33 2024-06-06T07:33:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/33 2024-06-06T07:33:31.538 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 29/33 2024-06-06T07:33:31.539 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:31.539 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-06T07:33:31.539 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T07:33:31.539 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T07:33:31.539 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:31.653 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 30/33 2024-06-06T07:33:31.679 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 30/33 2024-06-06T07:33:31.679 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:31.680 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-06T07:33:31.680 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T07:33:31.680 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T07:33:31.680 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:18.2.2-1413.g8a28a0ae.el9.x86_64 31/33 2024-06-06T07:33:32.144 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:18.2.2-1413.g8a28a0ae.el9.x86_64 31/33 2024-06-06T07:33:32.144 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:32.144 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-06T07:33:32.144 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T07:33:32.144 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T07:33:32.144 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:18.2.2-1413.g8a28a0ae.el9.x86_64 32/33 2024-06-06T07:33:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:18.2.2-1413.g8a28a0ae.el9.x86_64 32/33 2024-06-06T07:33:32.378 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:32.378 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-06T07:33:32.378 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T07:33:32.378 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T07:33:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:32.443 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:18.2.2-1413.g8a28a0ae.el9.x86_64 33/33 2024-06-06T07:33:32.998 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 30/33 2024-06-06T07:33:33.027 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 30/33 2024-06-06T07:33:33.027 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:33.027 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-06T07:33:33.027 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T07:33:33.027 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T07:33:33.027 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:33.466 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-2:18.2.2-1413.g8a28a0ae.el9.x86_64 31/33 2024-06-06T07:33:33.489 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-2:18.2.2-1413.g8a28a0ae.el9.x86_64 31/33 2024-06-06T07:33:33.489 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:33.489 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-06T07:33:33.490 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T07:33:33.490 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T07:33:33.490 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:33.708 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-2:18.2.2-1413.g8a28a0ae.el9.x86_64 32/33 2024-06-06T07:33:33.732 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-2:18.2.2-1413.g8a28a0ae.el9.x86_64 32/33 2024-06-06T07:33:33.732 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:33.732 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-06T07:33:33.732 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T07:33:33.732 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T07:33:33.732 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:33.796 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-2:18.2.2-1413.g8a28a0ae.el9.x86_64 33/33 2024-06-06T07:33:34.115 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-2:18.2.2-1413.g8a28a0ae.el9.x86_64 33/33 2024-06-06T07:33:34.115 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 3/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:18.2.2-1413.g8a28a0ae.el9.x86_64 4/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 5/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1413.g8a28a0ae.el9. 6/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-06T07:33:34.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-06T07:33:34.117 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-06T07:33:34.118 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-06T07:33:34.118 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-06T07:33:34.118 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-06T07:33:34.118 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-06T07:33:34.118 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-06T07:33:34.118 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-06T07:33:34.118 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-06T07:33:34.981 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply-3.11-14.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-06T07:33:34.982 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:34.983 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:33:35.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-base 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-2:18.2.2-1413.g8a28a0ae.el9.x86_64 33/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 3/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-2:18.2.2-1413.g8a28a0ae.el9.x86_64 4/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 5/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1413.g8a28a0ae.el9. 6/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-06T07:33:35.456 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-06T07:33:35.457 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-06T07:33:35.458 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-06T07:33:35.458 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-06T07:33:35.458 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-06T07:33:35.458 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-06T07:33:35.458 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-06T07:33:35.458 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-06T07:33:35.458 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-06T07:33:35.458 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-06T07:33:35.910 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:47 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:35.991 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:33:36.033 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:33:36.034 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-06-06T07:33:36.034 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:33:36.084 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install cephadm 2024-06-06T07:33:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-06T07:33:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:36.314 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:33:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-06T07:33:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply-3.11-14.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:36.316 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:33:36.647 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:48 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:36.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-base 2024-06-06T07:33:36.769 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:33:36.769 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-06-06T07:33:36.770 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:36.770 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:33:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 224 k 2024-06-06T07:33:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:36.770 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:33:36.770 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:36.770 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-06-06T07:33:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:36.771 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 224 k 2024-06-06T07:33:36.771 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 216 k 2024-06-06T07:33:36.771 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:33:37.127 INFO:teuthology.orchestra.run.smithi082.stdout:cephadm-18.2.2-1413.g8a28a0ae.el9.noarch.rpm 629 kB/s | 224 kB 00:00 2024-06-06T07:33:37.128 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:37.128 INFO:teuthology.orchestra.run.smithi082.stdout:Total 626 kB/s | 224 kB 00:00 2024-06-06T07:33:37.128 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:33:37.133 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:33:37.134 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:33:37.136 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:33:37.136 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:33:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:33:37.256 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:49 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:37.337 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-base-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:33:37.379 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:33:37.380 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-06-06T07:33:37.380 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:33:37.444 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install cephadm 2024-06-06T07:33:37.673 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/1 2024-06-06T07:33:37.755 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/1 2024-06-06T07:33:38.018 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:50 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/1 2024-06-06T07:33:38.141 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:33:38.142 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:38.142 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-06-06T07:33:38.142 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:38.142 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:33:38.142 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 224 k 2024-06-06T07:33:38.142 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:38.142 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:33:38.142 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:38.143 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-06-06T07:33:38.143 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:38.143 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 224 k 2024-06-06T07:33:38.143 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 216 k 2024-06-06T07:33:38.143 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:33:38.380 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/1 2024-06-06T07:33:38.380 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:38.380 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:33:38.380 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:38.381 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:38.381 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:33:38.491 INFO:teuthology.orchestra.run.smithi184.stdout:cephadm-18.2.2-1413.g8a28a0ae.el9.noarch.rpm 644 kB/s | 224 kB 00:00 2024-06-06T07:33:38.492 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:38.492 INFO:teuthology.orchestra.run.smithi184.stdout:Total 642 kB/s | 224 kB 00:00 2024-06-06T07:33:38.492 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:33:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:33:38.497 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:33:38.500 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:33:38.500 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:33:38.573 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:33:38.596 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-immutable-object-cache 2024-06-06T07:33:38.945 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/1 2024-06-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/1 2024-06-06T07:33:39.178 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:51 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:39.302 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:33:39.303 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:39.303 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-06-06T07:33:39.303 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:39.303 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:33:39.304 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 143 k 2024-06-06T07:33:39.304 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:39.304 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:33:39.304 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:39.304 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-06-06T07:33:39.304 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:39.304 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 143 k 2024-06-06T07:33:39.304 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 395 k 2024-06-06T07:33:39.304 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:33:39.366 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/1 2024-06-06T07:33:39.533 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-immutable-object-cache-18.2.2-1413.g8a28a0 625 kB/s | 143 kB 00:00 2024-06-06T07:33:39.533 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:39.534 INFO:teuthology.orchestra.run.smithi082.stdout:Total 620 kB/s | 143 kB 00:00 2024-06-06T07:33:39.534 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:33:39.542 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:33:39.542 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:33:39.591 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/1 2024-06-06T07:33:39.591 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:39.591 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:33:39.591 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:39.591 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:39.591 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:33:39.598 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:33:39.598 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:33:39.743 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:33:39.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-immutable-object-cache 2024-06-06T07:33:39.842 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1413.g8a28a0ae. 1/1 2024-06-06T07:33:39.867 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1413.g8a28a0ae. 1/1 2024-06-06T07:33:39.867 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:39.867 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-06T07:33:39.867 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:40.366 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:52 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:40.491 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:33:40.491 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:40.492 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-06-06T07:33:40.492 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:40.492 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:33:40.492 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 143 k 2024-06-06T07:33:40.492 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:40.492 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:33:40.492 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:40.492 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-06-06T07:33:40.492 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:40.493 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 143 k 2024-06-06T07:33:40.493 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 395 k 2024-06-06T07:33:40.493 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:33:40.715 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1413.g8a28a0ae. 1/1 2024-06-06T07:33:40.715 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:40.716 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:33:40.716 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:40.716 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:40.716 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:33:40.797 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-immutable-object-cache-18.2.2-1413.g8a28a0 470 kB/s | 143 kB 00:00 2024-06-06T07:33:40.798 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:40.798 INFO:teuthology.orchestra.run.smithi184.stdout:Total 467 kB/s | 143 kB 00:00 2024-06-06T07:33:40.798 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:33:40.806 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:33:40.806 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:33:40.863 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:33:40.863 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:33:40.946 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr 2024-06-06T07:33:41.002 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:33:41.082 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1413.g8a28a0ae. 1/1 2024-06-06T07:33:41.106 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1413.g8a28a0ae. 1/1 2024-06-06T07:33:41.107 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:33:41.107 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-06T07:33:41.107 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:41.533 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:53 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:41.616 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:33:41.661 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:33:41.662 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-06-06T07:33:41.662 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:33:41.724 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-dashboard 2024-06-06T07:33:42.034 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1413.g8a28a0ae. 1/1 2024-06-06T07:33:42.035 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:42.035 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:33:42.035 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:33:42.035 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:42.035 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:33:42.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr 2024-06-06T07:33:42.290 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:54 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:42.413 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:33:42.413 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:42.413 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-06-06T07:33:42.413 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 3.5 M 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 24 k 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 16 k 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout:Install 5 Packages 2024-06-06T07:33:42.414 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:42.415 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 3.8 M 2024-06-06T07:33:42.415 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 87 M 2024-06-06T07:33:42.415 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:33:42.837 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:54 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:42.886 INFO:teuthology.orchestra.run.smithi082.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1413.g8a2 123 kB/s | 24 kB 00:00 2024-06-06T07:33:42.911 INFO:teuthology.orchestra.run.smithi082.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1413.g8a28 73 kB/s | 16 kB 00:00 2024-06-06T07:33:42.921 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-mgr-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:33:42.965 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:33:42.967 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-06-06T07:33:42.967 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:33:42.995 INFO:teuthology.orchestra.run.smithi082.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 283 kB/s | 31 kB 00:00 2024-06-06T07:33:43.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-dashboard 2024-06-06T07:33:43.045 INFO:teuthology.orchestra.run.smithi082.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-06-06T07:33:43.170 INFO:teuthology.orchestra.run.smithi082.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1413.g8a28a0ae 7.3 MB/s | 3.5 MB 00:00 2024-06-06T07:33:43.174 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:43.174 INFO:teuthology.orchestra.run.smithi082.stdout:Total 5.0 MB/s | 3.8 MB 00:00 2024-06-06T07:33:43.174 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:33:43.181 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:33:43.181 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:33:43.242 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:33:43.243 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:33:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:33:43.599 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:55 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-06-06T07:33:43.667 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-06-06T07:33:43.695 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1413.g8a28a0ae.el9.n 3/5 2024-06-06T07:33:43.722 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:33:43.723 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:43.723 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 3.5 M 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 24 k 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 16 k 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:43.724 INFO:teuthology.orchestra.run.smithi184.stdout:Install 5 Packages 2024-06-06T07:33:43.725 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:43.725 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.8 M 2024-06-06T07:33:43.725 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 87 M 2024-06-06T07:33:43.725 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:33:43.985 INFO:teuthology.orchestra.run.smithi184.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1413.g8a2 128 kB/s | 24 kB 00:00 2024-06-06T07:33:44.043 INFO:teuthology.orchestra.run.smithi184.stdout:(2/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 528 kB/s | 31 kB 00:00 2024-06-06T07:33:44.077 INFO:teuthology.orchestra.run.smithi184.stdout:(3/5): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-06-06T07:33:44.094 INFO:teuthology.orchestra.run.smithi184.stdout:(4/5): ceph-prometheus-alerts-18.2.2-1413.g8a28 55 kB/s | 16 kB 00:00 2024-06-06T07:33:44.344 INFO:teuthology.orchestra.run.smithi184.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1413.g8a28a0ae 6.4 MB/s | 3.5 MB 00:00 2024-06-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:44.347 INFO:teuthology.orchestra.run.smithi184.stdout:Total 6.1 MB/s | 3.8 MB 00:00 2024-06-06T07:33:44.348 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:33:44.355 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:33:44.355 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:33:44.416 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:33:44.416 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:33:44.738 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:33:44.739 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1413.g8a28a0ae.el9. 4/5 2024-06-06T07:33:44.774 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1413.g8a28a0ae.el9.noarc 5/5 2024-06-06T07:33:44.809 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-06-06T07:33:44.841 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-06-06T07:33:44.869 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1413.g8a28a0ae.el9.n 3/5 2024-06-06T07:33:45.872 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1413.g8a28a0ae.el9.noarc 5/5 2024-06-06T07:33:45.872 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1413.g8a28a0ae.el9. 1/5 2024-06-06T07:33:45.872 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1413.g8a28a0ae.el9.noarc 2/5 2024-06-06T07:33:45.872 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1413.g8a28a0ae.el9.n 3/5 2024-06-06T07:33:45.873 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-06-06T07:33:45.901 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1413.g8a28a0ae.el9. 4/5 2024-06-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1413.g8a28a0ae.el9.noarc 5/5 2024-06-06T07:33:46.127 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-06-06T07:33:46.127 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:46.127 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:33:46.127 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:46.127 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:46.127 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:46.127 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-06T07:33:46.128 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-06T07:33:46.128 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:46.128 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:33:46.337 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-06T07:33:46.900 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:58 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:47.019 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1413.g8a28a0ae.el9.noarc 5/5 2024-06-06T07:33:47.019 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1413.g8a28a0ae.el9. 1/5 2024-06-06T07:33:47.019 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1413.g8a28a0ae.el9.noarc 2/5 2024-06-06T07:33:47.019 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1413.g8a28a0ae.el9.n 3/5 2024-06-06T07:33:47.019 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-06-06T07:33:47.020 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 7.4 M 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-06T07:33:47.022 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout:Install 12 Packages 2024-06-06T07:33:47.023 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:47.024 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 38 M 2024-06-06T07:33:47.024 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 209 M 2024-06-06T07:33:47.024 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:33:47.253 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-06-06T07:33:47.253 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:47.253 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:33:47.253 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:47.253 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:47.253 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:47.253 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-06T07:33:47.254 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-06T07:33:47.254 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:47.254 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:33:47.364 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:47.365 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:47.463 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-06T07:33:47.535 INFO:teuthology.orchestra.run.smithi082.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 782 kB/s | 195 kB 00:00 2024-06-06T07:33:47.536 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:47.586 INFO:teuthology.orchestra.run.smithi082.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-06-06T07:33:47.602 INFO:teuthology.orchestra.run.smithi082.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 488 kB/s | 33 kB 00:00 2024-06-06T07:33:47.603 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:47.603 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:47.661 INFO:teuthology.orchestra.run.smithi082.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 312 kB/s | 18 kB 00:00 2024-06-06T07:33:47.703 INFO:teuthology.orchestra.run.smithi082.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 842 kB/s | 35 kB 00:00 2024-06-06T07:33:47.795 INFO:teuthology.orchestra.run.smithi082.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.2-14 15 MB/s | 7.4 MB 00:00 2024-06-06T07:33:47.845 INFO:teuthology.orchestra.run.smithi082.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-06-06T07:33:47.971 INFO:teuthology.orchestra.run.smithi082.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-06-06T07:33:47.996 INFO:teuthology.orchestra.run.smithi082.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-06-06T07:33:48.021 INFO:teuthology.orchestra.run.smithi082.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.9 MB/s | 250 kB 00:00 2024-06-06T07:33:48.028 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:00 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:48.096 INFO:teuthology.orchestra.run.smithi082.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 12 MB/s | 4.7 MB 00:00 2024-06-06T07:33:48.151 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:33:48.152 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2024-06-06T07:33:48.152 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-06-06T07:33:48.152 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2024-06-06T07:33:48.152 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:33:48.152 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 7.4 M 2024-06-06T07:33:48.152 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-06T07:33:48.153 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:48.154 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:33:48.154 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2024-06-06T07:33:48.154 INFO:teuthology.orchestra.run.smithi184.stdout:Install 12 Packages 2024-06-06T07:33:48.154 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:48.155 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 38 M 2024-06-06T07:33:48.155 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 209 M 2024-06-06T07:33:48.155 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:33:48.327 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:48.328 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:48.380 INFO:teuthology.orchestra.run.smithi082.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-06-06T07:33:48.382 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:48.383 INFO:teuthology.orchestra.run.smithi082.stdout:Total 28 MB/s | 38 MB 00:01 2024-06-06T07:33:48.443 INFO:teuthology.orchestra.run.smithi184.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-06-06T07:33:48.447 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:48.494 INFO:teuthology.orchestra.run.smithi184.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-06-06T07:33:48.519 INFO:teuthology.orchestra.run.smithi184.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 433 kB/s | 33 kB 00:00 2024-06-06T07:33:48.519 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:48.522 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:33:48.561 INFO:teuthology.orchestra.run.smithi184.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-06-06T07:33:48.594 INFO:teuthology.orchestra.run.smithi184.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 35 kB 00:00 2024-06-06T07:33:48.661 INFO:teuthology.orchestra.run.smithi184.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 18 MB/s | 3.0 MB 00:00 2024-06-06T07:33:48.703 INFO:teuthology.orchestra.run.smithi184.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 5.8 MB/s | 244 kB 00:00 2024-06-06T07:33:48.733 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:33:48.760 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:33:48.760 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:33:48.812 INFO:teuthology.orchestra.run.smithi184.stdout:(8/12): ceph-mgr-diskprediction-local-18.2.2-14 13 MB/s | 7.4 MB 00:00 2024-06-06T07:33:48.879 INFO:teuthology.orchestra.run.smithi184.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-06-06T07:33:48.954 INFO:teuthology.orchestra.run.smithi184.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-06-06T07:33:49.021 INFO:teuthology.orchestra.run.smithi184.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.2 MB/s | 250 kB 00:00 2024-06-06T07:33:49.221 INFO:teuthology.orchestra.run.smithi184.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-06-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi184.stdout:Total 36 MB/s | 38 MB 00:01 2024-06-06T07:33:49.267 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:33:49.267 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:33:49.569 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:33:49.595 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:33:49.595 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:33:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:33:50.084 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:33:50.084 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:33:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-06T07:33:50.136 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-06T07:33:50.235 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-06T07:33:50.271 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-06T07:33:50.451 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-06T07:33:50.478 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-06T07:33:50.561 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-06T07:33:50.830 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:33:50.899 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-06T07:33:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-06T07:33:51.046 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-06T07:33:51.104 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-06T07:33:51.223 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-06T07:33:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-06T07:33:51.320 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-06T07:33:51.349 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-06T07:33:51.397 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-06T07:33:52.091 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-06T07:33:52.209 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-06T07:33:52.598 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T07:33:53.488 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T07:33:53.623 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-06T07:33:53.651 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1413.g8a28a 12/12 2024-06-06T07:33:54.507 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-06T07:33:54.528 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1413.g8a28a 12/12 2024-06-06T07:33:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1413.g8a28a 12/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1413.g8a28a 1/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T07:33:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:33:56.852 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-rook 2024-06-06T07:33:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1413.g8a28a 12/12 2024-06-06T07:33:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1413.g8a28a 1/12 2024-06-06T07:33:57.113 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-06T07:33:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-06T07:33:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-06T07:33:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-06T07:33:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-06T07:33:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-06T07:33:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-06T07:33:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-06T07:33:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T07:33:57.114 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-06T07:33:57.436 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:09 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:57.537 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-06T07:33:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:57.538 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:33:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:33:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-06T07:33:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-06T07:33:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-06T07:33:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-06T07:33:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-06T07:33:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-06T07:33:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-06T07:33:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-06T07:33:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-06T07:33:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-06T07:33:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-06T07:33:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:57.539 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:33:57.559 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:33:57.560 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:57.560 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-06-06T07:33:57.560 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:57.560 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:33:57.560 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 49 k 2024-06-06T07:33:57.560 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-06-06T07:33:57.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-06T07:33:57.561 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-06T07:33:57.562 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:57.562 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:33:57.562 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:33:57.562 INFO:teuthology.orchestra.run.smithi082.stdout:Install 13 Packages 2024-06-06T07:33:57.562 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:33:57.563 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 2.2 M 2024-06-06T07:33:57.563 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 26 M 2024-06-06T07:33:57.563 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:33:57.877 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:57.879 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:57.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-rook 2024-06-06T07:33:57.951 INFO:teuthology.orchestra.run.smithi082.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 166 kB/s | 26 kB 00:00 2024-06-06T07:33:57.976 INFO:teuthology.orchestra.run.smithi082.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 19 kB 00:00 2024-06-06T07:33:57.977 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:57.977 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-pyasn1-0.4.8-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.4.8-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:58.001 INFO:teuthology.orchestra.run.smithi082.stdout:(3/13): ceph-mgr-rook-18.2.2-1413.g8a28a0ae.el9 241 kB/s | 49 kB 00:00 2024-06-06T07:33:58.043 INFO:teuthology.orchestra.run.smithi082.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-06-06T07:33:58.069 INFO:teuthology.orchestra.run.smithi082.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.8 MB/s | 222 kB 00:00 2024-06-06T07:33:58.094 INFO:teuthology.orchestra.run.smithi082.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-06-06T07:33:58.119 INFO:teuthology.orchestra.run.smithi082.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 644 kB/s | 32 kB 00:00 2024-06-06T07:33:58.144 INFO:teuthology.orchestra.run.smithi082.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 299 kB/s | 15 kB 00:00 2024-06-06T07:33:58.178 INFO:teuthology.orchestra.run.smithi082.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-06-06T07:33:58.228 INFO:teuthology.orchestra.run.smithi082.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.8 MB/s | 206 kB 00:00 2024-06-06T07:33:58.254 INFO:teuthology.orchestra.run.smithi082.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-06-06T07:33:58.278 INFO:teuthology.orchestra.run.smithi082.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-06-06T07:33:58.312 INFO:teuthology.orchestra.run.smithi082.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-06-06T07:33:58.315 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:58.315 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.9 MB/s | 2.2 MB 00:00 2024-06-06T07:33:58.378 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:33:58.393 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:33:58.393 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:10 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:33:58.477 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:33:58.479 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:33:58.592 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-06-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:58.593 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 49 k 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-06T07:33:58.594 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi184.stdout:Install 13 Packages 2024-06-06T07:33:58.595 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.2 M 2024-06-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 26 M 2024-06-06T07:33:58.596 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:33:58.835 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:58.835 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:58.941 INFO:teuthology.orchestra.run.smithi184.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 139 kB/s | 26 kB 00:00 2024-06-06T07:33:58.967 INFO:teuthology.orchestra.run.smithi184.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 91 kB/s | 19 kB 00:00 2024-06-06T07:33:58.967 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:58.968 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-pyasn1-0.4.8-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.4.8-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:33:58.976 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:33:59.026 INFO:teuthology.orchestra.run.smithi184.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-06-06T07:33:59.051 INFO:teuthology.orchestra.run.smithi184.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-06-06T07:33:59.077 INFO:teuthology.orchestra.run.smithi184.stdout:(5/13): ceph-mgr-rook-18.2.2-1413.g8a28a0ae.el9 154 kB/s | 49 kB 00:00 2024-06-06T07:33:59.110 INFO:teuthology.orchestra.run.smithi184.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.3 MB/s | 279 kB 00:00 2024-06-06T07:33:59.135 INFO:teuthology.orchestra.run.smithi184.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 642 kB/s | 54 kB 00:00 2024-06-06T07:33:59.151 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-06T07:33:59.160 INFO:teuthology.orchestra.run.smithi184.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 301 kB/s | 15 kB 00:00 2024-06-06T07:33:59.210 INFO:teuthology.orchestra.run.smithi184.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-06-06T07:33:59.214 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-06T07:33:59.244 INFO:teuthology.orchestra.run.smithi184.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.9 MB/s | 59 kB 00:00 2024-06-06T07:33:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 1.5 MB/s | 206 kB 00:00 2024-06-06T07:33:59.275 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-06T07:33:59.294 INFO:teuthology.orchestra.run.smithi184.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-06-06T07:33:59.321 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-06T07:33:59.328 INFO:teuthology.orchestra.run.smithi184.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-06-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:33:59.332 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-06-06T07:33:59.362 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-06T07:33:59.396 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:33:59.411 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:33:59.411 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:33:59.457 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-06T07:33:59.497 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:33:59.499 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:33:59.567 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-06T07:33:59.623 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-06T07:33:59.994 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:34:00.159 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-06T07:34:00.221 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-06T07:34:00.277 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-06T07:34:00.314 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-06T07:34:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-06T07:34:00.355 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-06T07:34:00.396 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-06T07:34:00.431 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-06T07:34:00.458 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-06T07:34:00.479 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-06T07:34:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:18.2.2-1413.g8a28a0ae.el9.noarch 13/13 2024-06-06T07:34:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-06T07:34:00.623 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-06T07:34:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1413.g8a28a0ae.el9.noarch 13/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-06T07:34:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-06T07:34:01.346 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-06T07:34:01.387 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-06T07:34:01.421 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-06T07:34:01.469 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-06T07:34:01.499 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-rook-2:18.2.2-1413.g8a28a0ae.el9.noarch 13/13 2024-06-06T07:34:01.609 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-06T07:34:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-06T07:34:01.611 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-06T07:34:01.611 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-06T07:34:01.611 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-06T07:34:01.611 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:01.611 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:01.852 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-cephadm 2024-06-06T07:34:02.250 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1413.g8a28a0ae.el9.noarch 13/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-06T07:34:02.251 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-06T07:34:02.434 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:14 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:02.559 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:02.559 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 139 k 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:02.560 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:34:02.561 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:02.561 INFO:teuthology.orchestra.run.smithi082.stdout:Install 6 Packages 2024-06-06T07:34:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:02.561 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 7.0 M 2024-06-06T07:34:02.561 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 33 M 2024-06-06T07:34:02.561 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:34:02.604 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-06T07:34:02.605 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-06T07:34:02.606 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-06T07:34:02.606 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-06T07:34:02.606 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-06T07:34:02.606 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-06T07:34:02.606 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:02.606 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:02.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-cephadm 2024-06-06T07:34:02.991 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:34:02.992 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:34:03.157 INFO:teuthology.orchestra.run.smithi082.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-06-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi082.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 16 MB/s | 548 kB 00:00 2024-06-06T07:34:03.215 INFO:teuthology.orchestra.run.smithi082.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-06T07:34:03.241 INFO:teuthology.orchestra.run.smithi082.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-06-06T07:34:03.383 INFO:teuthology.orchestra.run.smithi082.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-06-06T07:34:03.408 INFO:teuthology.orchestra.run.smithi082.stdout:(6/6): ceph-mgr-cephadm-18.2.2-1413.g8a28a0ae.e 280 kB/s | 139 kB 00:00 2024-06-06T07:34:03.410 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:03.410 INFO:teuthology.orchestra.run.smithi082.stdout:Total 8.2 MB/s | 7.0 MB 00:00 2024-06-06T07:34:03.460 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:15 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:03.481 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:34:03.492 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:34:03.492 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:34:03.584 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:03.585 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:03.585 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-06-06T07:34:03.585 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:03.585 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:34:03.585 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1413.g8a28a0ae.el9 ceph-noarch 139 k 2024-06-06T07:34:03.585 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2024-06-06T07:34:03.586 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 7.0 M 2024-06-06T07:34:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 33 M 2024-06-06T07:34:03.587 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:34:03.589 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:34:03.590 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:34:03.890 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:34:04.039 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-06T07:34:04.084 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-06T07:34:04.127 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:34:04.129 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T07:34:04.288 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-06-06T07:34:04.314 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): ceph-mgr-cephadm-18.2.2-1413.g8a28a0ae.e 515 kB/s | 139 kB 00:00 2024-06-06T07:34:04.339 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-06T07:34:04.365 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-06-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-06-06T07:34:04.481 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-06-06T07:34:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:Total 7.8 MB/s | 7.0 MB 00:00 2024-06-06T07:34:04.556 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:34:04.567 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:34:04.567 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:34:04.594 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-06T07:34:04.663 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:34:04.663 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:34:04.679 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-06T07:34:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-06T07:34:04.797 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 6/6 2024-06-06T07:34:04.964 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:34:05.120 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-06T07:34:05.175 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-06T07:34:05.560 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 6/6 2024-06-06T07:34:05.561 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/6 2024-06-06T07:34:05.561 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-06T07:34:05.561 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-06T07:34:05.561 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-06T07:34:05.561 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-06T07:34:05.715 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-06T07:34:05.802 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-06T07:34:05.824 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-06T07:34:05.824 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:05.824 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:34:05.824 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:34:05.824 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-06T07:34:05.824 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-06T07:34:05.824 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-06T07:34:05.825 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-06T07:34:05.825 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-06T07:34:05.825 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:05.825 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:05.864 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-06T07:34:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 6/6 2024-06-06T07:34:06.063 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-fuse 2024-06-06T07:34:06.633 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:18 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:06.690 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 6/6 2024-06-06T07:34:06.690 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 1/6 2024-06-06T07:34:06.690 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-06T07:34:06.690 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-06T07:34:06.690 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-06T07:34:06.691 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-06T07:34:06.756 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:06.756 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:06.756 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 849 k 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout:Install 2 Packages 2024-06-06T07:34:06.757 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:06.758 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 929 k 2024-06-06T07:34:06.758 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 2.7 M 2024-06-06T07:34:06.758 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:34:06.932 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:34:06.942 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm-2:18.2.2-1413.g8a28a0ae.el9.noarch 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:07.018 INFO:teuthology.orchestra.run.smithi082.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 572 kB/s | 80 kB 00:00 2024-06-06T07:34:07.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-fuse 2024-06-06T07:34:07.193 INFO:teuthology.orchestra.run.smithi082.stdout:(2/2): ceph-fuse-18.2.2-1413.g8a28a0ae.el9.x86_ 2.6 MB/s | 849 kB 00:00 2024-06-06T07:34:07.195 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:07.195 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.1 MB/s | 929 kB 00:00 2024-06-06T07:34:07.207 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:34:07.228 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:34:07.228 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:34:07.305 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:34:07.306 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:34:07.499 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:34:07.643 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-06T07:34:07.710 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/2 2024-06-06T07:34:07.754 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:19 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:07.878 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:07.879 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:07.879 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-06-06T07:34:07.879 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:07.879 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:34:07.879 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 849 k 2024-06-06T07:34:07.879 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-06-06T07:34:07.879 INFO:teuthology.orchestra.run.smithi184.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-06T07:34:07.879 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:07.879 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:34:07.880 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:07.880 INFO:teuthology.orchestra.run.smithi184.stdout:Install 2 Packages 2024-06-06T07:34:07.880 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:07.880 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 929 k 2024-06-06T07:34:07.880 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 2.7 M 2024-06-06T07:34:07.880 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:34:08.038 INFO:teuthology.orchestra.run.smithi184.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T07:34:08.150 INFO:teuthology.orchestra.run.smithi184.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 546 kB/s | 80 kB 00:00 2024-06-06T07:34:08.346 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/2 2024-06-06T07:34:08.346 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/2 2024-06-06T07:34:08.433 INFO:teuthology.orchestra.run.smithi184.stdout:(2/2): ceph-fuse-18.2.2-1413.g8a28a0ae.el9.x86_ 1.9 MB/s | 849 kB 00:00 2024-06-06T07:34:08.434 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:08.435 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.6 MB/s | 929 kB 00:00 2024-06-06T07:34:08.447 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:34:08.546 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:34:08.547 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:34:08.617 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-06T07:34:08.617 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:08.618 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:34:08.618 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-06T07:34:08.618 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:08.618 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:08.735 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:34:08.871 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados-devel 2024-06-06T07:34:08.908 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-06T07:34:08.973 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/2 2024-06-06T07:34:09.464 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:21 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:09.591 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:09.591 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-06-06T07:34:09.592 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:09.592 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:34:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 127 k 2024-06-06T07:34:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:09.592 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:34:09.592 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:09.592 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-06-06T07:34:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:09.593 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 127 k 2024-06-06T07:34:09.593 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 456 k 2024-06-06T07:34:09.593 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:34:09.617 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2/2 2024-06-06T07:34:09.617 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/2 2024-06-06T07:34:09.824 INFO:teuthology.orchestra.run.smithi082.stdout:librados-devel-18.2.2-1413.g8a28a0ae.el9.x86_64 550 kB/s | 127 kB 00:00 2024-06-06T07:34:09.824 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:09.824 INFO:teuthology.orchestra.run.smithi082.stdout:Total 546 kB/s | 127 kB 00:00 2024-06-06T07:34:09.824 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:34:09.830 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:34:09.830 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:34:09.869 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-06T07:34:09.869 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:09.869 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:34:09.870 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-06T07:34:09.870 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:09.870 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:09.892 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:34:09.892 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:34:10.041 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:34:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:10.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados-devel 2024-06-06T07:34:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:10.728 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:22 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:10.807 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:34:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:10.807 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:10.858 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:10.858 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:10.858 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-06-06T07:34:10.859 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:10.859 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:34:10.859 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 127 k 2024-06-06T07:34:10.859 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:10.859 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:34:10.859 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:10.859 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-06-06T07:34:10.859 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 127 k 2024-06-06T07:34:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 456 k 2024-06-06T07:34:10.860 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:34:11.058 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs2 2024-06-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi184.stdout:librados-devel-18.2.2-1413.g8a28a0ae.el9.x86_64 359 kB/s | 127 kB 00:00 2024-06-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi184.stdout:Total 357 kB/s | 127 kB 00:00 2024-06-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:34:11.219 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:34:11.219 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:34:11.281 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:34:11.281 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:34:11.443 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:34:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:11.675 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:23 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:11.760 INFO:teuthology.orchestra.run.smithi082.stdout:Package libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:11.802 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-06-06T07:34:11.803 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:11.870 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs-devel 2024-06-06T07:34:11.976 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:12.225 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:12.225 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:12.225 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:34:12.225 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:12.225 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:12.226 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:12.439 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:24 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:12.489 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs2 2024-06-06T07:34:12.560 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 31 k 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 31 k 2024-06-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 139 k 2024-06-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:34:12.736 INFO:teuthology.orchestra.run.smithi082.stdout:libcephfs-devel-18.2.2-1413.g8a28a0ae.el9.x86_6 178 kB/s | 31 kB 00:00 2024-06-06T07:34:12.736 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:12.737 INFO:teuthology.orchestra.run.smithi082.stdout:Total 177 kB/s | 31 kB 00:00 2024-06-06T07:34:12.737 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:34:12.741 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:34:12.742 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:34:12.754 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:34:12.754 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:34:12.838 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:34:12.941 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:25 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:13.174 INFO:teuthology.orchestra.run.smithi184.stdout:Package libcephfs2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:13.220 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:13.221 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-06-06T07:34:13.221 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:13.286 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs-devel 2024-06-06T07:34:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:13.540 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:34:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:13.540 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:13.743 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados2 2024-06-06T07:34:13.859 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:25 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:13.982 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:13.983 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:13.983 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-06-06T07:34:13.983 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:13.983 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:34:13.983 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 31 k 2024-06-06T07:34:13.983 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:13.983 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:34:13.984 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:13.984 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-06-06T07:34:13.984 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:13.984 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 31 k 2024-06-06T07:34:13.984 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 139 k 2024-06-06T07:34:13.984 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:34:14.149 INFO:teuthology.orchestra.run.smithi184.stdout:libcephfs-devel-18.2.2-1413.g8a28a0ae.el9.x86_6 188 kB/s | 31 kB 00:00 2024-06-06T07:34:14.149 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:14.150 INFO:teuthology.orchestra.run.smithi184.stdout:Total 187 kB/s | 31 kB 00:00 2024-06-06T07:34:14.150 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:34:14.153 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:34:14.153 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:34:14.172 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:34:14.172 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:34:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:34:14.315 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:26 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:14.346 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:14.395 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:14.437 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:14.438 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-06-06T07:34:14.438 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:14.491 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librbd1 2024-06-06T07:34:14.718 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:14.982 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:14.982 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:14.982 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:34:14.983 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:14.983 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:14.983 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:15.062 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:27 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:15.143 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:15.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados2 2024-06-06T07:34:15.185 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:15.186 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-06-06T07:34:15.186 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:15.250 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rados 2024-06-06T07:34:15.738 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:27 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:15.819 INFO:teuthology.orchestra.run.smithi184.stdout:Package librados2-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:15.822 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:27 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:15.861 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:15.862 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-06-06T07:34:15.862 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:15.904 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rados-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:15.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librbd1 2024-06-06T07:34:15.946 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:15.947 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-06-06T07:34:15.947 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:16.007 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rgw 2024-06-06T07:34:16.494 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:28 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:16.575 INFO:teuthology.orchestra.run.smithi184.stdout:Package librbd1-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:16.578 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:28 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:16.616 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:16.618 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-06-06T07:34:16.618 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:16.661 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:16.693 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rados 2024-06-06T07:34:16.702 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:16.703 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-06-06T07:34:16.703 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:16.765 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-cephfs 2024-06-06T07:34:17.274 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:29 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:17.339 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:29 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:17.357 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rados-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:17.399 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:17.400 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-06-06T07:34:17.400 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:17.419 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-cephfs-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:17.460 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:17.461 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-06-06T07:34:17.461 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:17.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rgw 2024-06-06T07:34:17.515 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rbd 2024-06-06T07:34:18.055 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:30 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:18.088 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:30 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:18.137 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rgw-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:18.170 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:18.178 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:18.179 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-06-06T07:34:18.180 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:18.212 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:18.213 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-06-06T07:34:18.213 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:18.243 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-cephfs 2024-06-06T07:34:18.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-fuse 2024-06-06T07:34:18.819 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:30 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:30 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:18.900 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-cephfs-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:18.941 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:18.942 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-06-06T07:34:18.943 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:18.967 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 87 k 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-06-06T07:34:18.968 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:18.969 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 87 k 2024-06-06T07:34:18.969 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 227 k 2024-06-06T07:34:18.969 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:34:19.018 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rbd 2024-06-06T07:34:19.197 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-fuse-18.2.2-1413.g8a28a0ae.el9.x86_64.rpm 380 kB/s | 87 kB 00:00 2024-06-06T07:34:19.198 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:19.198 INFO:teuthology.orchestra.run.smithi082.stdout:Total 378 kB/s | 87 kB 00:00 2024-06-06T07:34:19.198 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:34:19.205 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:34:19.205 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:34:19.267 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:34:19.268 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:34:19.416 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:34:19.502 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:19.594 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:31 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:19.676 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 is already installed. 2024-06-06T07:34:19.719 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:19.720 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-06-06T07:34:19.720 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:19.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-fuse 2024-06-06T07:34:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:20.106 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:34:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:20.106 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:20.374 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-mirror 2024-06-06T07:34:20.374 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:32 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:20.498 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:20.499 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:20.499 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-06-06T07:34:20.499 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:20.499 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:34:20.499 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 87 k 2024-06-06T07:34:20.499 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:20.499 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:34:20.499 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:20.499 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-06-06T07:34:20.500 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:20.500 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 87 k 2024-06-06T07:34:20.500 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 227 k 2024-06-06T07:34:20.500 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-fuse-18.2.2-1413.g8a28a0ae.el9.x86_64.rpm 396 kB/s | 87 kB 00:00 2024-06-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi184.stdout:Total 394 kB/s | 87 kB 00:00 2024-06-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:34:20.790 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:34:20.790 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:34:20.952 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:34:20.960 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:32 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:21.056 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:21.087 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:21.088 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:21.088 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-06-06T07:34:21.088 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:21.088 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:34:21.088 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 3.0 M 2024-06-06T07:34:21.088 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:21.088 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:34:21.088 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:21.088 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-06-06T07:34:21.089 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:21.089 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 3.0 M 2024-06-06T07:34:21.089 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 12 M 2024-06-06T07:34:21.089 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:34:21.432 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:21.459 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-mirror-18.2.2-1413.g8a28a0ae.el9.x86_64.rpm 8.1 MB/s | 3.0 MB 00:00 2024-06-06T07:34:21.460 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:21.460 INFO:teuthology.orchestra.run.smithi082.stdout:Total 8.0 MB/s | 3.0 MB 00:00 2024-06-06T07:34:21.461 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:34:21.469 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:34:21.469 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:34:21.547 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:34:21.548 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:34:21.673 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:21.673 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:21.673 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:34:21.673 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:21.673 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:21.673 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:21.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-mirror 2024-06-06T07:34:21.960 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:34:22.035 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:22.060 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:22.060 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:34:22.060 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-06T07:34:22.060 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T07:34:22.060 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T07:34:22.060 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:22.504 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:34 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:22.630 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:22.631 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:22.631 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-06-06T07:34:22.631 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:22.632 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:34:22.632 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 3.0 M 2024-06-06T07:34:22.632 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:22.632 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:34:22.632 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:22.632 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-06-06T07:34:22.632 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:22.632 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.0 M 2024-06-06T07:34:22.633 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 12 M 2024-06-06T07:34:22.633 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-mirror-18.2.2-1413.g8a28a0ae.el9.x86_64.rpm 8.6 MB/s | 3.0 MB 00:00 2024-06-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:22.981 INFO:teuthology.orchestra.run.smithi184.stdout:Total 8.5 MB/s | 3.0 MB 00:00 2024-06-06T07:34:22.981 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:34:22.990 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:34:22.990 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:34:23.045 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:23.046 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:23.046 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:34:23.046 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:23.046 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:23.046 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:23.068 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:34:23.069 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:34:23.287 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-nbd 2024-06-06T07:34:23.481 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:34:23.547 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:23.573 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:23.573 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T07:34:23.573 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-06T07:34:23.573 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T07:34:23.573 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T07:34:23.573 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:23.877 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:35 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:24.000 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-06-06T07:34:24.000 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 172 k 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 172 k 2024-06-06T07:34:24.001 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 494 k 2024-06-06T07:34:24.002 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-06-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-nbd-18.2.2-1413.g8a28a0ae.el9.x86_64.rpm 782 kB/s | 172 kB 00:00 2024-06-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:Total 776 kB/s | 172 kB 00:00 2024-06-06T07:34:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-06-06T07:34:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-06-06T07:34:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-06-06T07:34:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-06-06T07:34:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-06-06T07:34:24.449 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-06-06T07:34:24.553 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:24.614 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:24.615 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:24.615 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:34:24.615 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:24.615 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:24.615 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:24.883 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-nbd 2024-06-06T07:34:24.948 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:25.221 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:25.221 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:25.221 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-06-06T07:34:25.221 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:25.221 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:34:25.221 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-06-06T07:34:25.459 DEBUG:teuthology.parallel:result is None 2024-06-06T07:34:25.469 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:37 ago on Thu 06 Jun 2024 07:32:48 AM UTC. 2024-06-06T07:34:25.593 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-06-06T07:34:25.618 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:25.618 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-06-06T07:34:25.618 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:25.618 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-06-06T07:34:25.618 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd x86_64 2:18.2.2-1413.g8a28a0ae.el9 ceph 172 k 2024-06-06T07:34:25.618 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:25.618 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-06-06T07:34:25.618 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-06-06T07:34:25.619 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-06-06T07:34:25.619 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:25.619 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 172 k 2024-06-06T07:34:25.619 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 494 k 2024-06-06T07:34:25.619 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-06-06T07:34:25.975 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-nbd-18.2.2-1413.g8a28a0ae.el9.x86_64.rpm 453 kB/s | 172 kB 00:00 2024-06-06T07:34:25.976 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:34:25.976 INFO:teuthology.orchestra.run.smithi184.stdout:Total 451 kB/s | 172 kB 00:00 2024-06-06T07:34:25.976 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-06-06T07:34:25.983 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-06-06T07:34:25.983 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-06-06T07:34:26.046 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-06-06T07:34:26.047 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-06-06T07:34:26.194 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-06-06T07:34:26.298 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:26.990 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 1/1 2024-06-06T07:34:26.991 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:26.991 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-06-06T07:34:26.991 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-2:18.2.2-1413.g8a28a0ae.el9.x86_64 2024-06-06T07:34:26.991 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:34:26.991 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-06-06T07:34:27.253 DEBUG:teuthology.parallel:result is None 2024-06-06T07:34:27.254 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T07:34:27.254 INFO:teuthology.packaging:ref: None 2024-06-06T07:34:27.254 INFO:teuthology.packaging:tag: None 2024-06-06T07:34:27.254 INFO:teuthology.packaging:branch: reef 2024-06-06T07:34:27.254 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T07:34:27.254 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-06-06T07:34:27.816 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-06T07:34:27.841 INFO:teuthology.orchestra.run.smithi082.stdout:18.2.2-1413.g8a28a0ae.el9 2024-06-06T07:34:27.842 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1413.g8a28a0ae.el9 2024-06-06T07:34:27.842 INFO:teuthology.task.install:The correct ceph version 18.2.2-1413.g8a28a0ae is installed. 2024-06-06T07:34:27.843 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T07:34:27.843 INFO:teuthology.packaging:ref: None 2024-06-06T07:34:27.844 INFO:teuthology.packaging:tag: None 2024-06-06T07:34:27.844 INFO:teuthology.packaging:branch: reef 2024-06-06T07:34:27.844 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T07:34:27.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-06-06T07:34:28.409 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-06T07:34:28.431 INFO:teuthology.orchestra.run.smithi184.stdout:18.2.2-1413.g8a28a0ae.el9 2024-06-06T07:34:28.431 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1413.g8a28a0ae.el9 2024-06-06T07:34:28.432 INFO:teuthology.task.install:The correct ceph version 18.2.2-1413.g8a28a0ae is installed. 2024-06-06T07:34:28.433 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-06T07:34:28.434 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:34:28.434 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-06T07:34:28.464 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:34:28.464 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-06T07:34:28.497 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-06T07:34:28.498 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:34:28.498 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-06-06T07:34:28.533 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-06T07:34:28.601 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:34:28.601 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2024-06-06T07:34:28.626 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-06T07:34:28.691 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-06T07:34:28.691 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:34:28.691 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-06T07:34:28.718 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-06T07:34:28.785 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:34:28.785 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-06T07:34:28.810 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-06T07:34:28.874 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-06T07:34:28.874 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:34:28.874 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-06-06T07:34:28.901 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-06T07:34:28.967 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:34:28.967 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2024-06-06T07:34:28.992 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-06T07:34:29.057 INFO:teuthology.run_tasks:Running task print... 2024-06-06T07:34:29.067 INFO:teuthology.task.print:**** done install task... 2024-06-06T07:34:29.067 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-06T07:34:29.198 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-06T07:34:29.198 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-06-06T07:34:29.199 INFO:tasks.cephadm:Cluster fsid is 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:34:29.199 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-06T07:34:29.199 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-06-06T07:34:29.199 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi082': '172.21.15.82', 'mon.smithi184': '172.21.15.184'} 2024-06-06T07:34:29.199 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-06T07:34:29.200 DEBUG:teuthology.orchestra.run.smithi082:> sudo hostname $(hostname -s) 2024-06-06T07:34:29.227 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname $(hostname -s) 2024-06-06T07:34:29.251 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-06-06T07:34:29.252 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T07:34:29.395 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55273/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2946-g3d89accf', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2946.g3d89accf'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-30 17:04:17.570799', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-06-06T07:34:29.901 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 8a28a0aebfa0e95f142cb00d67403340afc840a5 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-06-06T07:34:29.902 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/x86_64/flavors/default/cephadm 2024-06-06T07:34:29.903 DEBUG:teuthology.orchestra.run.smithi082:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-06T07:34:30.462 INFO:teuthology.orchestra.run.smithi082.stdout:-rw-r--r--. 1 ubuntu ubuntu 215331 Jun 6 07:34 /home/ubuntu/cephtest/cephadm 2024-06-06T07:34:30.462 DEBUG:teuthology.orchestra.run.smithi184:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/8a28a0aebfa0e95f142cb00d67403340afc840a5/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-06T07:34:31.044 INFO:teuthology.orchestra.run.smithi184.stdout:-rw-r--r--. 1 ubuntu ubuntu 215331 Jun 6 07:34 /home/ubuntu/cephtest/cephadm 2024-06-06T07:34:31.044 DEBUG:teuthology.orchestra.run.smithi082:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-06T07:34:31.064 DEBUG:teuthology.orchestra.run.smithi184:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-06T07:34:31.095 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-06-06T07:34:31.095 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-06-06T07:34:31.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-06-06T07:34:31.988 INFO:teuthology.orchestra.run.smithi082.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-06-06T07:34:32.008 INFO:teuthology.orchestra.run.smithi184.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-06-06T07:35:15.938 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-06-06T07:35:15.939 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_version": "ceph version 18.2.2-1413-g8a28a0ae (8a28a0aebfa0e95f142cb00d67403340afc840a5) reef (stable)", 2024-06-06T07:35:15.939 INFO:teuthology.orchestra.run.smithi082.stdout: "image_id": "27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb", 2024-06-06T07:35:15.939 INFO:teuthology.orchestra.run.smithi082.stdout: "repo_digests": [ 2024-06-06T07:35:15.939 INFO:teuthology.orchestra.run.smithi082.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6" 2024-06-06T07:35:15.939 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-06-06T07:35:15.939 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-06-06T07:35:19.640 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-06-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 18.2.2-1413-g8a28a0ae (8a28a0aebfa0e95f142cb00d67403340afc840a5) reef (stable)", 2024-06-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi184.stdout: "image_id": "27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb", 2024-06-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi184.stdout: "repo_digests": [ 2024-06-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi184.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6" 2024-06-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-06-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-06-06T07:35:19.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph 2024-06-06T07:35:19.693 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph 2024-06-06T07:35:19.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /etc/ceph 2024-06-06T07:35:19.762 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /etc/ceph 2024-06-06T07:35:19.787 INFO:tasks.cephadm:Writing seed config... 2024-06-06T07:35:19.788 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-06T07:35:19.788 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-06T07:35:19.789 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-06-06T07:35:19.790 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-06T07:35:19.790 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-06T07:35:19.790 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-06T07:35:19.790 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-06T07:35:19.790 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-06T07:35:19.790 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-06T07:35:19.790 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-06T07:35:19.790 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-06T07:35:19.791 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-06T07:35:19.792 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-06T07:35:19.792 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-06T07:35:19.792 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-06T07:35:19.792 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-06T07:35:19.792 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-06T07:35:19.792 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-06T07:35:19.792 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-06T07:35:19.792 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-06T07:35:19.792 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-06T07:35:19.793 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:35:19.793 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-06T07:35:19.819 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 355cdb60-23d7-11ef-bc9b-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-06T07:35:19.819 DEBUG:teuthology.orchestra.run.smithi082:mon.smithi082> sudo journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi082.service 2024-06-06T07:35:19.862 INFO:tasks.cephadm:Bootstrapping... 2024-06-06T07:35:19.862 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.82 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-06T07:35:20.020 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-06-06T07:35:20.020 INFO:teuthology.orchestra.run.smithi082.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '355cdb60-23d7-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.82', '--skip-admin-label'] 2024-06-06T07:35:20.045 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stdout 5.0.2 2024-06-06T07:35:20.045 INFO:teuthology.orchestra.run.smithi082.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-06T07:35:20.046 INFO:teuthology.orchestra.run.smithi082.stdout:Verifying podman|docker is present... 2024-06-06T07:35:20.068 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stdout 5.0.2 2024-06-06T07:35:20.068 INFO:teuthology.orchestra.run.smithi082.stdout:Verifying lvm2 is present... 2024-06-06T07:35:20.069 INFO:teuthology.orchestra.run.smithi082.stdout:Verifying time synchronization is in place... 2024-06-06T07:35:20.076 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-06T07:35:20.076 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-06T07:35:20.082 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-06T07:35:20.082 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout inactive 2024-06-06T07:35:20.088 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout enabled 2024-06-06T07:35:20.095 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout active 2024-06-06T07:35:20.095 INFO:teuthology.orchestra.run.smithi082.stdout:Unit chronyd.service is enabled and running 2024-06-06T07:35:20.095 INFO:teuthology.orchestra.run.smithi082.stdout:Repeating the final host check... 2024-06-06T07:35:20.117 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stdout 5.0.2 2024-06-06T07:35:20.118 INFO:teuthology.orchestra.run.smithi082.stdout:podman (/bin/podman) version 5.0.2 is present 2024-06-06T07:35:20.118 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl is present 2024-06-06T07:35:20.118 INFO:teuthology.orchestra.run.smithi082.stdout:lvcreate is present 2024-06-06T07:35:20.126 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-06T07:35:20.126 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-06T07:35:20.130 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-06T07:35:20.130 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout inactive 2024-06-06T07:35:20.136 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout enabled 2024-06-06T07:35:20.143 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout active 2024-06-06T07:35:20.143 INFO:teuthology.orchestra.run.smithi082.stdout:Unit chronyd.service is enabled and running 2024-06-06T07:35:20.143 INFO:teuthology.orchestra.run.smithi082.stdout:Host looks OK 2024-06-06T07:35:20.143 INFO:teuthology.orchestra.run.smithi082.stdout:Cluster fsid: 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:35:20.143 INFO:teuthology.orchestra.run.smithi082.stdout:Acquiring lock 140463442673968 on /run/cephadm/355cdb60-23d7-11ef-bc9b-c7b262605968.lock 2024-06-06T07:35:20.143 INFO:teuthology.orchestra.run.smithi082.stdout:Lock 140463442673968 acquired on /run/cephadm/355cdb60-23d7-11ef-bc9b-c7b262605968.lock 2024-06-06T07:35:20.144 INFO:teuthology.orchestra.run.smithi082.stdout:Verifying IP 172.21.15.82 port 3300 ... 2024-06-06T07:35:20.144 INFO:teuthology.orchestra.run.smithi082.stdout:Verifying IP 172.21.15.82 port 6789 ... 2024-06-06T07:35:20.144 INFO:teuthology.orchestra.run.smithi082.stdout:Base mon IP(s) is [172.21.15.82:3300, 172.21.15.82:6789], mon addrv is [v2:172.21.15.82:3300,v1:172.21.15.82:6789] 2024-06-06T07:35:20.147 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.82 metric 100 2024-06-06T07:35:20.147 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.82 metric 100 2024-06-06T07:35:20.150 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-06T07:35:20.150 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-06T07:35:20.150 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1770sec hoplimit 64 pref medium 2024-06-06T07:35:20.152 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-06T07:35:20.152 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-06T07:35:20.152 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-06T07:35:20.152 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-06T07:35:20.152 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d3ff/64 scope link 2024-06-06T07:35:20.152 INFO:teuthology.orchestra.run.smithi082.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-06T07:35:20.153 INFO:teuthology.orchestra.run.smithi082.stdout:Mon IP `172.21.15.82` is in CIDR network `172.21.0.0/20` 2024-06-06T07:35:20.153 INFO:teuthology.orchestra.run.smithi082.stdout:Mon IP `172.21.15.82` is in CIDR network `172.21.0.0/20` 2024-06-06T07:35:20.153 INFO:teuthology.orchestra.run.smithi082.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-06T07:35:20.154 INFO:teuthology.orchestra.run.smithi082.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-06T07:35:20.154 INFO:teuthology.orchestra.run.smithi082.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-06-06T07:35:21.125 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stdout 27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb 2024-06-06T07:35:21.125 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-06-06T07:35:21.125 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stderr Getting image source signatures 2024-06-06T07:35:21.125 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stderr Copying blob sha256:ca62abcd7c345b5fb51de283a08184c9e803329bf7cc6a864a2a194aed123f95 2024-06-06T07:35:21.125 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stderr Copying blob sha256:36850128dbaa55a7fd38bf12640a70ab57c121fdc76af02f72b423e059979691 2024-06-06T07:35:21.126 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stderr Copying config sha256:27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb 2024-06-06T07:35:21.126 INFO:teuthology.orchestra.run.smithi082.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-06T07:35:22.029 INFO:teuthology.orchestra.run.smithi082.stdout:ceph: stdout ceph version 18.2.2-1413-g8a28a0ae (8a28a0aebfa0e95f142cb00d67403340afc840a5) reef (stable) 2024-06-06T07:35:22.029 INFO:teuthology.orchestra.run.smithi082.stdout:Ceph version: ceph version 18.2.2-1413-g8a28a0ae (8a28a0aebfa0e95f142cb00d67403340afc840a5) reef (stable) 2024-06-06T07:35:22.029 INFO:teuthology.orchestra.run.smithi082.stdout:Extracting ceph user uid/gid from container image... 2024-06-06T07:35:22.892 INFO:teuthology.orchestra.run.smithi082.stdout:stat: stdout 167 167 2024-06-06T07:35:22.892 INFO:teuthology.orchestra.run.smithi082.stdout:Creating initial keys... 2024-06-06T07:35:23.731 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph-authtool: stdout AQC7ZmFmvOxhDhAAJ7W/KXuEA2tW9tDaK7cLUA== 2024-06-06T07:35:24.670 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph-authtool: stdout AQC8ZmFmET9GAxAA/QnEW1b2YQiP+gFomS0wRw== 2024-06-06T07:35:25.550 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph-authtool: stdout AQC8ZmFmG6DCOhAAt7PzCWKprsy8ZlFD2s/2kQ== 2024-06-06T07:35:25.551 INFO:teuthology.orchestra.run.smithi082.stdout:Creating initial monmap... 2024-06-06T07:35:26.416 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-06T07:35:26.416 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-06-06T07:35:26.416 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:35:26.416 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-06T07:35:26.416 INFO:teuthology.orchestra.run.smithi082.stdout:monmaptool for smithi082 [v2:172.21.15.82:3300,v1:172.21.15.82:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-06T07:35:26.417 INFO:teuthology.orchestra.run.smithi082.stdout:setting min_mon_release = pacific 2024-06-06T07:35:26.417 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/monmaptool: set fsid to 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:35:26.417 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-06T07:35:26.417 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:35:26.417 INFO:teuthology.orchestra.run.smithi082.stdout:Creating mon... 2024-06-06T07:35:27.321 INFO:teuthology.orchestra.run.smithi082.stdout:create mon.smithi082 on 2024-06-06T07:35:27.523 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-06T07:35:27.681 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-06T07:35:27.855 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968.target. 2024-06-06T07:35:27.855 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968.target. 2024-06-06T07:35:28.096 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi082 2024-06-06T07:35:28.096 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Failed to reset failed state of unit ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi082.service: Unit ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi082.service not loaded. 2024-06-06T07:35:28.264 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968.target.wants/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi082.service → /etc/systemd/system/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@.service. 2024-06-06T07:35:28.823 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T07:35:28.824 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout disabled 2024-06-06T07:35:28.831 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T07:35:28.831 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout inactive 2024-06-06T07:35:28.831 INFO:teuthology.orchestra.run.smithi082.stdout:firewalld.service is not enabled 2024-06-06T07:35:28.831 INFO:teuthology.orchestra.run.smithi082.stdout:Not possible to enable service . firewalld.service is not available 2024-06-06T07:35:28.832 INFO:teuthology.orchestra.run.smithi082.stdout:Waiting for mon to start... 2024-06-06T07:35:28.832 INFO:teuthology.orchestra.run.smithi082.stdout:Waiting for mon... 2024-06-06T07:35:28.880 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:28 smithi082 systemd[1]: Started Ceph mon.smithi082 for 355cdb60-23d7-11ef-bc9b-c7b262605968. 2024-06-06T07:35:29.847 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:29 smithi082 ceph-mon[28983]: mon.smithi082 is new leader, mons smithi082 in quorum (ranks 0) 2024-06-06T07:35:29.848 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:29 smithi082 ceph-mon[28983]: monmap e1: 1 mons at {smithi082=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T07:35:29.848 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:29 smithi082 ceph-mon[28983]: fsmap 2024-06-06T07:35:29.848 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:29 smithi082 ceph-mon[28983]: osdmap e1: 0 total, 0 up, 0 in 2024-06-06T07:35:29.848 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:29 smithi082 ceph-mon[28983]: mgrmap e1: no daemons active 2024-06-06T07:35:29.848 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:29 smithi082 ceph-mon[28983]: from='client.? 172.21.15.82:0/511612' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-06T07:35:29.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout cluster: 2024-06-06T07:35:29.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout id: 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:35:29.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-06T07:35:29.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:29.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout services: 2024-06-06T07:35:29.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi082 (age 0.551809s) 2024-06-06T07:35:29.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout data: 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout pgs: 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.236+0000 7f487d64f640 1 Processor -- start 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.236+0000 7f487d64f640 1 -- start start 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.237+0000 7f487d64f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:29.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.237+0000 7f487d64f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4878079e40 con 0x7f48781066f0 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.237+0000 7f4876ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.237+0000 7f4876ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47706/0 (socket says 172.21.15.82:47706) 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.237+0000 7f4876ffd640 1 -- 172.21.15.82:0/4023795021 learned_addr learned my addr 172.21.15.82:0/4023795021 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.238+0000 7f4876ffd640 1 -- 172.21.15.82:0/4023795021 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4878109020 con 0x7f48781066f0 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.239+0000 7f4876ffd640 1 --2- 172.21.15.82:0/4023795021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878108ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4868009820 tx=0x7f486802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3f5796f2250c22 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f4875ffb640 1 -- 172.21.15.82:0/4023795021 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f486802ad40 con 0x7f48781066f0 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f4875ffb640 1 -- 172.21.15.82:0/4023795021 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4868032440 con 0x7f48781066f0 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f4875ffb640 1 -- 172.21.15.82:0/4023795021 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f486802aea0 con 0x7f48781066f0 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f487d64f640 1 -- 172.21.15.82:0/4023795021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 msgr2=0x7f4878108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f487d64f640 1 --2- 172.21.15.82:0/4023795021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878108ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4868009820 tx=0x7f486802a270 comp rx=0 tx=0).stop 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f487d64f640 1 -- 172.21.15.82:0/4023795021 shutdown_connections 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f487d64f640 1 --2- 172.21.15.82:0/4023795021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878108ae0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:29.858 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f487d64f640 1 -- 172.21.15.82:0/4023795021 >> 172.21.15.82:0/4023795021 conn(0x7f48781000e0 msgr2=0x7f4878102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f487d64f640 1 -- 172.21.15.82:0/4023795021 shutdown_connections 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.240+0000 7f487d64f640 1 -- 172.21.15.82:0/4023795021 wait complete. 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.241+0000 7f487d64f640 1 Processor -- start 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.241+0000 7f487d64f640 1 -- start start 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.241+0000 7f487d64f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.241+0000 7f487d64f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4878199650 con 0x7f48781066f0 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.241+0000 7f4876ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.241+0000 7f4876ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47718/0 (socket says 172.21.15.82:47718) 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.241+0000 7f4876ffd640 1 -- 172.21.15.82:0/511612 learned_addr learned my addr 172.21.15.82:0/511612 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.242+0000 7f4876ffd640 1 -- 172.21.15.82:0/511612 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680094d0 con 0x7f48781066f0 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.242+0000 7f4876ffd640 1 --2- 172.21.15.82:0/511612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878199110 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f48680090b0 tx=0x7f48680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:29.859 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.242+0000 7f485ffff640 1 -- 172.21.15.82:0/511612 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48680308d0 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.242+0000 7f485ffff640 1 -- 172.21.15.82:0/511612 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4868030a30 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.242+0000 7f485ffff640 1 -- 172.21.15.82:0/511612 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4868030d30 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.242+0000 7f487d64f640 1 -- 172.21.15.82:0/511612 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4878199850 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.242+0000 7f487d64f640 1 -- 172.21.15.82:0/511612 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4878199cf0 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.242+0000 7f485ffff640 1 -- 172.21.15.82:0/511612 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f4868039040 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.242+0000 7f485ffff640 1 -- 172.21.15.82:0/511612 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f486803b8c0 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.243+0000 7f487d64f640 1 -- 172.21.15.82:0/511612 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4878192f80 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.245+0000 7f485ffff640 1 -- 172.21.15.82:0/511612 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f4868037070 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.306+0000 7f487d64f640 1 -- 172.21.15.82:0/511612 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f487802c480 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.307+0000 7f485ffff640 1 -- 172.21.15.82:0/511612 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f486803ba80 con 0x7f48781066f0 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.307+0000 7f487d64f640 1 -- 172.21.15.82:0/511612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 msgr2=0x7f4878199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:29.860 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.307+0000 7f487d64f640 1 --2- 172.21.15.82:0/511612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878199110 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f48680090b0 tx=0x7f48680046c0 comp rx=0 tx=0).stop 2024-06-06T07:35:29.861 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.307+0000 7f487d64f640 1 -- 172.21.15.82:0/511612 shutdown_connections 2024-06-06T07:35:29.861 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.307+0000 7f487d64f640 1 --2- 172.21.15.82:0/511612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48781066f0 0x7f4878199110 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:29.861 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.307+0000 7f487d64f640 1 -- 172.21.15.82:0/511612 >> 172.21.15.82:0/511612 conn(0x7f48781000e0 msgr2=0x7f4878102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:29.861 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.308+0000 7f487d64f640 1 -- 172.21.15.82:0/511612 shutdown_connections 2024-06-06T07:35:29.861 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:29.308+0000 7f487d64f640 1 -- 172.21.15.82:0/511612 wait complete. 2024-06-06T07:35:29.861 INFO:teuthology.orchestra.run.smithi082.stdout:mon is available 2024-06-06T07:35:29.861 INFO:teuthology.orchestra.run.smithi082.stdout:Assimilating anything we can from ceph.conf... 2024-06-06T07:35:30.870 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:30 smithi082 ceph-mon[28983]: from='client.? 172.21.15.82:0/45888135' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-06T07:35:30.870 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:30 smithi082 ceph-mon[28983]: from='client.? 172.21.15.82:0/45888135' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-06T07:35:30.879 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:30.879 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout [global] 2024-06-06T07:35:30.879 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout fsid = 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:35:30.879 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.82:3300,v1:172.21.15.82:6789] 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout [osd] 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-06T07:35:30.880 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.217+0000 7fbb31968640 1 Processor -- start 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.217+0000 7fbb31968640 1 -- start start 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.217+0000 7fbb31968640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.217+0000 7fbb31968640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c079e40 con 0x7fbb2c1066f0 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.217+0000 7fbb2affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.217+0000 7fbb2affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47732/0 (socket says 172.21.15.82:47732) 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.217+0000 7fbb2affd640 1 -- 172.21.15.82:0/3521554944 learned_addr learned my addr 172.21.15.82:0/3521554944 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.218+0000 7fbb2affd640 1 -- 172.21.15.82:0/3521554944 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c109020 con 0x7fbb2c1066f0 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.218+0000 7fbb2affd640 1 --2- 172.21.15.82:0/3521554944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c108ae0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009820 tx=0x7fbb1c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e8cf81eb7edf20 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb2a7fc640 1 -- 172.21.15.82:0/3521554944 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1c02ad40 con 0x7fbb2c1066f0 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb2a7fc640 1 -- 172.21.15.82:0/3521554944 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c032440 con 0x7fbb2c1066f0 2024-06-06T07:35:30.881 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb2a7fc640 1 -- 172.21.15.82:0/3521554944 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1c02aea0 con 0x7fbb2c1066f0 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb31968640 1 -- 172.21.15.82:0/3521554944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 msgr2=0x7fbb2c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb31968640 1 --2- 172.21.15.82:0/3521554944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c108ae0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009820 tx=0x7fbb1c02a270 comp rx=0 tx=0).stop 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb31968640 1 -- 172.21.15.82:0/3521554944 shutdown_connections 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb31968640 1 --2- 172.21.15.82:0/3521554944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c108ae0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb31968640 1 -- 172.21.15.82:0/3521554944 >> 172.21.15.82:0/3521554944 conn(0x7fbb2c1000e0 msgr2=0x7fbb2c102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb31968640 1 -- 172.21.15.82:0/3521554944 shutdown_connections 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.219+0000 7fbb31968640 1 -- 172.21.15.82:0/3521554944 wait complete. 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.220+0000 7fbb31968640 1 Processor -- start 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.220+0000 7fbb31968640 1 -- start start 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.220+0000 7fbb31968640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c105f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.220+0000 7fbb31968640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c106460 con 0x7fbb2c1066f0 2024-06-06T07:35:30.882 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.220+0000 7fbb2affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.220+0000 7fbb2affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47742/0 (socket says 172.21.15.82:47742) 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.220+0000 7fbb2affd640 1 -- 172.21.15.82:0/45888135 learned_addr learned my addr 172.21.15.82:0/45888135 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.220+0000 7fbb2affd640 1 -- 172.21.15.82:0/45888135 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c0094d0 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.221+0000 7fbb2affd640 1 --2- 172.21.15.82:0/45888135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c105f20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0090b0 tx=0x7fbb1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.221+0000 7fbb30966640 1 -- 172.21.15.82:0/45888135 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1c0308d0 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.221+0000 7fbb30966640 1 -- 172.21.15.82:0/45888135 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c030a30 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.221+0000 7fbb31968640 1 -- 172.21.15.82:0/45888135 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2c1046a0 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.221+0000 7fbb30966640 1 -- 172.21.15.82:0/45888135 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1c030d30 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.221+0000 7fbb31968640 1 -- 172.21.15.82:0/45888135 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2c104b40 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.221+0000 7fbb30966640 1 -- 172.21.15.82:0/45888135 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fbb1c039040 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.221+0000 7fbb30966640 1 -- 172.21.15.82:0/45888135 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbb1c03b880 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.222+0000 7fbb31968640 1 -- 172.21.15.82:0/45888135 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb2c061ae0 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.224+0000 7fbb30966640 1 -- 172.21.15.82:0/45888135 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fbb1c037070 con 0x7fbb2c1066f0 2024-06-06T07:35:30.883 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.283+0000 7fbb31968640 1 -- 172.21.15.82:0/45888135 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fbb2c1092d0 con 0x7fbb2c1066f0 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.286+0000 7fbb30966640 1 -- 172.21.15.82:0/45888135 <== mon.0 v2:172.21.15.82:3300/0 7 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbb1c046d00 con 0x7fbb2c1066f0 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.286+0000 7fbb30966640 1 -- 172.21.15.82:0/45888135 <== mon.0 v2:172.21.15.82:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fbb1c045d30 con 0x7fbb2c1066f0 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.287+0000 7fbb31968640 1 -- 172.21.15.82:0/45888135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 msgr2=0x7fbb2c105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.287+0000 7fbb31968640 1 --2- 172.21.15.82:0/45888135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c105f20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0090b0 tx=0x7fbb1c0046c0 comp rx=0 tx=0).stop 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.287+0000 7fbb31968640 1 -- 172.21.15.82:0/45888135 shutdown_connections 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.287+0000 7fbb31968640 1 --2- 172.21.15.82:0/45888135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb2c1066f0 0x7fbb2c105f20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.287+0000 7fbb31968640 1 -- 172.21.15.82:0/45888135 >> 172.21.15.82:0/45888135 conn(0x7fbb2c1000e0 msgr2=0x7fbb2c102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.288+0000 7fbb31968640 1 -- 172.21.15.82:0/45888135 shutdown_connections 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:30.288+0000 7fbb31968640 1 -- 172.21.15.82:0/45888135 wait complete. 2024-06-06T07:35:30.884 INFO:teuthology.orchestra.run.smithi082.stdout:Generating new minimal ceph.conf... 2024-06-06T07:35:32.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:31 smithi082 ceph-mon[28983]: from='client.? 172.21.15.82:0/893853700' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:35:36.893 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.276+0000 7ff0c1800640 1 Processor -- start 2024-06-06T07:35:36.893 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.276+0000 7ff0c1800640 1 -- start start 2024-06-06T07:35:36.893 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.276+0000 7ff0c1800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:36.893 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.276+0000 7ff0c1800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc106630 con 0x7ff0bc105c40 2024-06-06T07:35:36.893 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.276+0000 7ff0baffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:36.893 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.276+0000 7ff0baffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47744/0 (socket says 172.21.15.82:47744) 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.276+0000 7ff0baffd640 1 -- 172.21.15.82:0/570257839 learned_addr learned my addr 172.21.15.82:0/570257839 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.276+0000 7ff0baffd640 1 -- 172.21.15.82:0/570257839 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc106e40 con 0x7ff0bc105c40 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.277+0000 7ff0baffd640 1 --2- 172.21.15.82:0/570257839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc106060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac009820 tx=0x7ff0ac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb1cce1673e556c server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.277+0000 7ff0ba7fc640 1 -- 172.21.15.82:0/570257839 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ac02ad40 con 0x7ff0bc105c40 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.277+0000 7ff0ba7fc640 1 -- 172.21.15.82:0/570257839 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff0ac0328c0 con 0x7ff0bc105c40 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.277+0000 7ff0ba7fc640 1 -- 172.21.15.82:0/570257839 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ac0307f0 con 0x7ff0bc105c40 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.278+0000 7ff0c1800640 1 -- 172.21.15.82:0/570257839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 msgr2=0x7ff0bc106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.278+0000 7ff0c1800640 1 --2- 172.21.15.82:0/570257839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc106060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac009820 tx=0x7ff0ac02a270 comp rx=0 tx=0).stop 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.278+0000 7ff0c1800640 1 -- 172.21.15.82:0/570257839 shutdown_connections 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.278+0000 7ff0c1800640 1 --2- 172.21.15.82:0/570257839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc106060 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.278+0000 7ff0c1800640 1 -- 172.21.15.82:0/570257839 >> 172.21.15.82:0/570257839 conn(0x7ff0bc1011c0 msgr2=0x7ff0bc103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.278+0000 7ff0c1800640 1 -- 172.21.15.82:0/570257839 shutdown_connections 2024-06-06T07:35:36.894 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.278+0000 7ff0c1800640 1 -- 172.21.15.82:0/570257839 wait complete. 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.278+0000 7ff0c1800640 1 Processor -- start 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0c1800640 1 -- start start 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0c1800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0c1800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc199db0 con 0x7ff0bc105c40 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0baffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0baffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47756/0 (socket says 172.21.15.82:47756) 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0baffd640 1 -- 172.21.15.82:0/893853700 learned_addr learned my addr 172.21.15.82:0/893853700 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0baffd640 1 -- 172.21.15.82:0/893853700 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac0094d0 con 0x7ff0bc105c40 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0baffd640 1 --2- 172.21.15.82:0/893853700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc199870 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac030d30 tx=0x7ff0ac030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0b8ff9640 1 -- 172.21.15.82:0/893853700 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ac032c40 con 0x7ff0bc105c40 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0b8ff9640 1 -- 172.21.15.82:0/893853700 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff0ac032da0 con 0x7ff0bc105c40 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0c1800640 1 -- 172.21.15.82:0/893853700 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0bc199fb0 con 0x7ff0bc105c40 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0b8ff9640 1 -- 172.21.15.82:0/893853700 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ac03c6a0 con 0x7ff0bc105c40 2024-06-06T07:35:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.279+0000 7ff0c1800640 1 -- 172.21.15.82:0/893853700 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0bc19a3d0 con 0x7ff0bc105c40 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.280+0000 7ff0b8ff9640 1 -- 172.21.15.82:0/893853700 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7ff0ac03cb00 con 0x7ff0bc105c40 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.280+0000 7ff0b8ff9640 1 -- 172.21.15.82:0/893853700 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff0ac0463c0 con 0x7ff0bc105c40 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.280+0000 7ff0c1800640 1 -- 172.21.15.82:0/893853700 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0bc061ae0 con 0x7ff0bc105c40 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.283+0000 7ff0b8ff9640 1 -- 172.21.15.82:0/893853700 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7ff0ac037070 con 0x7ff0bc105c40 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.343+0000 7ff0c1800640 1 -- 172.21.15.82:0/893853700 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff0bc10a070 con 0x7ff0bc105c40 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.343+0000 7ff0b8ff9640 1 -- 172.21.15.82:0/893853700 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7ff0ac03da90 con 0x7ff0bc105c40 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.344+0000 7ff0c1800640 1 -- 172.21.15.82:0/893853700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 msgr2=0x7ff0bc199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.344+0000 7ff0c1800640 1 --2- 172.21.15.82:0/893853700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc199870 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac030d30 tx=0x7ff0ac030d60 comp rx=0 tx=0).stop 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.344+0000 7ff0c1800640 1 -- 172.21.15.82:0/893853700 shutdown_connections 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.344+0000 7ff0c1800640 1 --2- 172.21.15.82:0/893853700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0bc105c40 0x7ff0bc199870 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.344+0000 7ff0c1800640 1 -- 172.21.15.82:0/893853700 >> 172.21.15.82:0/893853700 conn(0x7ff0bc1011c0 msgr2=0x7ff0bc101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.344+0000 7ff0c1800640 1 -- 172.21.15.82:0/893853700 shutdown_connections 2024-06-06T07:35:36.896 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:31.344+0000 7ff0c1800640 1 -- 172.21.15.82:0/893853700 wait complete. 2024-06-06T07:35:36.897 INFO:teuthology.orchestra.run.smithi082.stdout:Restarting the monitor... 2024-06-06T07:35:37.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:36 smithi082 systemd[1]: Stopping Ceph mon.smithi082 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:35:38.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:37 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082[28979]: 2024-06-06T07:35:37.581+0000 7f4cd8376640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi082 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T07:35:38.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:37 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082[28979]: 2024-06-06T07:35:37.581+0000 7f4cd8376640 -1 mon.smithi082@0(leader) e1 *** Got Signal Terminated *** 2024-06-06T07:35:38.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:37 smithi082 podman[29227]: 2024-06-06 07:35:37.827845707 +0000 UTC m=+0.762214781 container died 90c0e44dc9221615ef0052ca70d1e033ab94b765ab91e71601ef4b12e67648b3 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082, RELEASE=reef-8a28a0a, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-06T07:35:38.340 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 podman[29227]: 2024-06-06 07:35:38.122158698 +0000 UTC m=+1.056527765 container cleanup 90c0e44dc9221615ef0052ca70d1e033ab94b765ab91e71601ef4b12e67648b3 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-06T07:35:38.340 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 bash[29227]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082 2024-06-06T07:35:38.678 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 podman[29241]: 2024-06-06 07:35:38.339847594 +0000 UTC m=+0.504799277 container remove 90c0e44dc9221615ef0052ca70d1e033ab94b765ab91e71601ef4b12e67648b3 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, RELEASE=reef-8a28a0a, GIT_CLEAN=True) 2024-06-06T07:35:38.678 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 systemd[1]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi082.service: Deactivated successfully. 2024-06-06T07:35:38.678 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 systemd[1]: Stopped Ceph mon.smithi082 for 355cdb60-23d7-11ef-bc9b-c7b262605968. 2024-06-06T07:35:38.679 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 systemd[1]: Starting Ceph mon.smithi082 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:35:38.935 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 podman[29334]: 2024-06-06 07:35:38.58163249 +0000 UTC m=+0.017524952 image pull 27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb quay.ceph.io/ceph-ci/ceph:reef 2024-06-06T07:35:38.935 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 podman[29334]: 2024-06-06 07:35:38.68221953 +0000 UTC m=+0.118111993 container create 5b5c9653fb28e96948315a57aaa2a2b3518bdf7778c0592333d5beea466142f0 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=reef-8a28a0a, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-06T07:35:38.935 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 podman[29334]: 2024-06-06 07:35:38.868780029 +0000 UTC m=+0.304672487 container init 5b5c9653fb28e96948315a57aaa2a2b3518bdf7778c0592333d5beea466142f0 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082, ceph=True, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-8a28a0a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2) 2024-06-06T07:35:38.935 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 podman[29334]: 2024-06-06 07:35:38.872000722 +0000 UTC m=+0.307893180 container start 5b5c9653fb28e96948315a57aaa2a2b3518bdf7778c0592333d5beea466142f0 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=reef-8a28a0a, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T07:35:38.935 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: set uid:gid to 167:167 (ceph:ceph) 2024-06-06T07:35:38.935 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: ceph version 18.2.2-1413-g8a28a0ae (8a28a0aebfa0e95f142cb00d67403340afc840a5) reef (stable), process ceph-mon, pid 2 2024-06-06T07:35:38.935 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: pidfile_write: ignore empty --pid-file 2024-06-06T07:35:38.935 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: load: jerasure load: lrc 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: RocksDB version: 7.9.2 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Git sha 0 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Compile date 2024-06-05 15:25:41 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: DB SUMMARY 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: DB Session ID: XLXR3E6K2CI9QIW756TH 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: CURRENT file: CURRENT 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: IDENTITY file: IDENTITY 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi082/store.db dir, Total Num: 1, files: 000008.sst 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi082/store.db: 000009.log size: 98453 ; 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.error_if_exists: 0 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.create_if_missing: 0 2024-06-06T07:35:38.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.paranoid_checks: 1 2024-06-06T07:35:38.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-06T07:35:38.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-06T07:35:38.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-06T07:35:38.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.env: 0x55b8da25dba0 2024-06-06T07:35:38.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.fs: PosixFileSystem 2024-06-06T07:35:38.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.info_log: 0x55b8db14cdc0 2024-06-06T07:35:38.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_file_opening_threads: 16 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.statistics: (nil) 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.use_fsync: 0 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_log_file_size: 0 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.keep_log_file_num: 1000 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.recycle_log_file_num: 0 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.allow_fallocate: 1 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.allow_mmap_reads: 0 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.allow_mmap_writes: 0 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.use_direct_reads: 0 2024-06-06T07:35:38.938 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.create_missing_column_families: 0 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.db_log_dir: 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.wal_dir: 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.advise_random_on_open: 1 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.db_write_buffer_size: 0 2024-06-06T07:35:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.write_buffer_manager: 0x55b8db11e140 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.rate_limiter: (nil) 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.wal_recovery_mode: 2 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.enable_thread_tracking: 0 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.enable_pipelined_write: 0 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.unordered_write: 0 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-06T07:35:38.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.row_cache: None 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.wal_filter: None 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.allow_ingest_behind: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.two_write_queues: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.manual_wal_flush: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.wal_compression: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.atomic_flush: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.log_readahead_size: 0 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-06T07:35:38.941 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.best_efforts_recovery: 0 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.allow_data_in_errors: 0 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.db_host_id: __hostname__ 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_background_jobs: 2 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_background_compactions: -1 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_subcompactions: 1 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_total_wal_size: 0 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-06T07:35:38.942 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_open_files: -1 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bytes_per_sync: 0 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_readahead_size: 0 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_background_flushes: -1 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Compression algorithms supported: 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: kZSTD supported: 0 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: kXpressCompression supported: 0 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: kBZip2Compression supported: 0 2024-06-06T07:35:38.943 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: kLZ4Compression supported: 1 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: kZlibCompression supported: 1 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: kLZ4HCCompression supported: 1 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: kSnappyCompression supported: 1 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi082/store.db/MANIFEST-000010 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.merge_operator: 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_filter: None 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_filter_factory: None 2024-06-06T07:35:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.sst_partitioner_factory: None 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b8db14ca60) 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: cache_index_and_filter_blocks: 1 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: pin_top_level_index_and_filter: 1 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: index_type: 0 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: data_block_index_type: 0 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: index_shortening: 1 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: checksum: 4 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: no_block_cache: 0 2024-06-06T07:35:38.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: block_cache: 0x55b8db143350 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: block_cache_name: BinnedLRUCache 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: block_cache_options: 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: capacity : 536870912 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: num_shard_bits : 4 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: strict_capacity_limit : 0 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: high_pri_pool_ratio: 0.000 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: block_cache_compressed: (nil) 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: persistent_cache: (nil) 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: block_size: 4096 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: block_size_deviation: 10 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: block_restart_interval: 16 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: index_block_restart_interval: 1 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: metadata_block_size: 4096 2024-06-06T07:35:38.946 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: partition_filters: 0 2024-06-06T07:35:38.947 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: use_delta_encoding: 1 2024-06-06T07:35:38.947 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: filter_policy: bloomfilter 2024-06-06T07:35:38.947 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: whole_key_filtering: 1 2024-06-06T07:35:38.947 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: verify_compression: 0 2024-06-06T07:35:38.947 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: read_amp_bytes_per_bit: 0 2024-06-06T07:35:38.947 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: format_version: 5 2024-06-06T07:35:38.947 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: enable_index_compression: 1 2024-06-06T07:35:38.947 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: block_align: 0 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: max_auto_readahead_size: 262144 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: prepopulate_block_cache: 0 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: initial_auto_readahead_size: 8192 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: num_file_reads_for_auto_readahead: 2 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.write_buffer_size: 33554432 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_write_buffer_number: 2 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression: NoCompression 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression: Disabled 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.prefix_extractor: nullptr 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.num_levels: 7 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-06T07:35:38.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression_opts.level: 32767 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression_opts.strategy: 0 2024-06-06T07:35:38.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression_opts.enabled: false 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.target_file_size_base: 67108864 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-06T07:35:38.950 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.arena_block_size: 1048576 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.disable_auto_compactions: 0 2024-06-06T07:35:38.951 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.table_properties_collectors: 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.inplace_update_support: 0 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-06T07:35:38.952 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.bloom_locality: 0 2024-06-06T07:35:38.953 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.max_successive_merges: 0 2024-06-06T07:35:38.953 INFO:teuthology.orchestra.run.smithi082.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-06T07:35:38.953 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-06T07:35:38.953 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.paranoid_file_checks: 0 2024-06-06T07:35:38.953 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.force_consistency_checks: 1 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.report_bg_io_stats: 0 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.ttl: 2592000 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.enable_blob_files: false 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.min_blob_size: 0 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.blob_file_size: 268435456 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-06T07:35:38.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.blob_file_starting_level: 0 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi082/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4e55f23f-bf84-490e-bba8-d265828ec9b7 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717659338897959, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717659338898974, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93730, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91764, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84328, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717659338, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4e55f23f-bf84-490e-bba8-d265828ec9b7", "db_session_id": "XLXR3E6K2CI9QIW756TH", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717659338899072, "job": 1, "event": "recovery_finished"} 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi082/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b8db16ce00 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: DB pointer 0x55b8db260000 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-06T07:35:38.955 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: ** DB Stats ** 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: ** Compaction Stats [default] ** 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: L0 2/0 93.41 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Sum 2/0 93.41 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-06T07:35:38.956 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: ** Compaction Stats [default] ** 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-06T07:35:38.957 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Cumulative compaction: 0.00 GB write, 28.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Interval compaction: 0.00 GB write, 28.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Block cache BinnedLRUCache@0x55b8db143350#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: starting mon.smithi082 rank 0 at public addrs [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] at bind addrs [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi082 fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mon.smithi082@-1(???) e1 preinit fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mon.smithi082@-1(???).mds e1 new map 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mon.smithi082@-1(???).mds e1 print_map 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: e1 2024-06-06T07:35:38.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: legacy client fscid: -1 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: No filesystems configured 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mon.smithi082@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mon.smithi082@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mon.smithi082@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mon.smithi082@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mon.smithi082@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 bash[29334]: 5b5c9653fb28e96948315a57aaa2a2b3518bdf7778c0592333d5beea466142f0 2024-06-06T07:35:38.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 systemd[1]: Started Ceph mon.smithi082 for 355cdb60-23d7-11ef-bc9b-c7b262605968. 2024-06-06T07:35:39.246 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mon.smithi082 is new leader, mons smithi082 in quorum (ranks 0) 2024-06-06T07:35:39.246 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: monmap e1: 1 mons at {smithi082=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T07:35:39.246 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: fsmap 2024-06-06T07:35:39.246 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: osdmap e1: 0 total, 0 up, 0 in 2024-06-06T07:35:39.246 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:38 smithi082 ceph-mon[29354]: mgrmap e1: no daemons active 2024-06-06T07:35:39.986 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.325+0000 7fa533310640 1 Processor -- start 2024-06-06T07:35:39.986 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.325+0000 7fa533310640 1 -- start start 2024-06-06T07:35:39.986 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.325+0000 7fa533310640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c108b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:39.986 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.326+0000 7fa533310640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c079dc0 con 0x7fa52c106770 2024-06-06T07:35:39.986 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.326+0000 7fa5310ac640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c108b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.326+0000 7fa5310ac640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c108b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42744/0 (socket says 172.21.15.82:42744) 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.326+0000 7fa5310ac640 1 -- 172.21.15.82:0/1885436865 learned_addr learned my addr 172.21.15.82:0/1885436865 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.326+0000 7fa5310ac640 1 -- 172.21.15.82:0/1885436865 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c1090a0 con 0x7fa52c106770 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.328+0000 7fa5310ac640 1 --2- 172.21.15.82:0/1885436865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c108b60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa528009e80 tx=0x7fa52802a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73dc3072aa7257af server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.328+0000 7fa523fff640 1 -- 172.21.15.82:0/1885436865 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa528031400 con 0x7fa52c106770 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.328+0000 7fa523fff640 1 -- 172.21.15.82:0/1885436865 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa52802d050 con 0x7fa52c106770 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.328+0000 7fa523fff640 1 -- 172.21.15.82:0/1885436865 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa528007910 con 0x7fa52c106770 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.329+0000 7fa533310640 1 -- 172.21.15.82:0/1885436865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 msgr2=0x7fa52c108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.329+0000 7fa533310640 1 --2- 172.21.15.82:0/1885436865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c108b60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa528009e80 tx=0x7fa52802a7a0 comp rx=0 tx=0).stop 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.329+0000 7fa533310640 1 -- 172.21.15.82:0/1885436865 shutdown_connections 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.329+0000 7fa533310640 1 --2- 172.21.15.82:0/1885436865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c108b60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.329+0000 7fa533310640 1 -- 172.21.15.82:0/1885436865 >> 172.21.15.82:0/1885436865 conn(0x7fa52c100120 msgr2=0x7fa52c102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.329+0000 7fa533310640 1 -- 172.21.15.82:0/1885436865 shutdown_connections 2024-06-06T07:35:39.987 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.329+0000 7fa533310640 1 -- 172.21.15.82:0/1885436865 wait complete. 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.329+0000 7fa533310640 1 Processor -- start 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.329+0000 7fa533310640 1 -- start start 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa533310640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c1998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa533310640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c199e10 con 0x7fa52c106770 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa5310ac640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c1998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa5310ac640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c1998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42750/0 (socket says 172.21.15.82:42750) 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa5310ac640 1 -- 172.21.15.82:0/741972193 learned_addr learned my addr 172.21.15.82:0/741972193 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa5310ac640 1 -- 172.21.15.82:0/741972193 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa528009b30 con 0x7fa52c106770 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa5310ac640 1 --2- 172.21.15.82:0/741972193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c1998d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa5280099e0 tx=0x7fa528031610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa5227fc640 1 -- 172.21.15.82:0/741972193 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa528031bb0 con 0x7fa52c106770 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa5227fc640 1 -- 172.21.15.82:0/741972193 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa528031d10 con 0x7fa52c106770 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa533310640 1 -- 172.21.15.82:0/741972193 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c19a010 con 0x7fa52c106770 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa5227fc640 1 -- 172.21.15.82:0/741972193 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa52803a720 con 0x7fa52c106770 2024-06-06T07:35:39.988 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.330+0000 7fa533310640 1 -- 172.21.15.82:0/741972193 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c19a510 con 0x7fa52c106770 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.331+0000 7fa5227fc640 1 -- 172.21.15.82:0/741972193 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fa528038040 con 0x7fa52c106770 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.331+0000 7fa5227fc640 1 -- 172.21.15.82:0/741972193 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa528044cb0 con 0x7fa52c106770 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.331+0000 7fa533310640 1 -- 172.21.15.82:0/741972193 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c109d70 con 0x7fa52c106770 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.334+0000 7fa5227fc640 1 -- 172.21.15.82:0/741972193 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fa528049060 con 0x7fa52c106770 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.393+0000 7fa533310640 1 -- 172.21.15.82:0/741972193 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fa52c19ce00 con 0x7fa52c106770 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.398+0000 7fa5227fc640 1 -- 172.21.15.82:0/741972193 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa528040070 con 0x7fa52c106770 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.398+0000 7fa533310640 1 -- 172.21.15.82:0/741972193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 msgr2=0x7fa52c1998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.398+0000 7fa533310640 1 --2- 172.21.15.82:0/741972193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c1998d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa5280099e0 tx=0x7fa528031610 comp rx=0 tx=0).stop 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.398+0000 7fa533310640 1 -- 172.21.15.82:0/741972193 shutdown_connections 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.398+0000 7fa533310640 1 --2- 172.21.15.82:0/741972193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa52c106770 0x7fa52c1998d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.398+0000 7fa533310640 1 -- 172.21.15.82:0/741972193 >> 172.21.15.82:0/741972193 conn(0x7fa52c100120 msgr2=0x7fa52c100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.399+0000 7fa533310640 1 -- 172.21.15.82:0/741972193 shutdown_connections 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:39.399+0000 7fa533310640 1 -- 172.21.15.82:0/741972193 wait complete. 2024-06-06T07:35:39.989 INFO:teuthology.orchestra.run.smithi082.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-06T07:35:40.009 INFO:teuthology.orchestra.run.smithi082.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-06T07:35:40.009 INFO:teuthology.orchestra.run.smithi082.stdout:Creating mgr... 2024-06-06T07:35:40.009 INFO:teuthology.orchestra.run.smithi082.stdout:Verifying port 0.0.0.0:9283 ... 2024-06-06T07:35:40.010 INFO:teuthology.orchestra.run.smithi082.stdout:Verifying port 0.0.0.0:8765 ... 2024-06-06T07:35:40.010 INFO:teuthology.orchestra.run.smithi082.stdout:Verifying port 0.0.0.0:8443 ... 2024-06-06T07:35:40.262 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mgr.smithi082.kocfig 2024-06-06T07:35:40.263 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Failed to reset failed state of unit ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mgr.smithi082.kocfig.service: Unit ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mgr.smithi082.kocfig.service not loaded. 2024-06-06T07:35:40.409 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968.target.wants/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mgr.smithi082.kocfig.service → /etc/systemd/system/ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@.service. 2024-06-06T07:35:40.652 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:40 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/741972193' entity='client.admin' 2024-06-06T07:35:40.885 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T07:35:40.885 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout disabled 2024-06-06T07:35:40.892 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T07:35:40.893 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout inactive 2024-06-06T07:35:40.893 INFO:teuthology.orchestra.run.smithi082.stdout:firewalld.service is not enabled 2024-06-06T07:35:40.893 INFO:teuthology.orchestra.run.smithi082.stdout:Not possible to enable service . firewalld.service is not available 2024-06-06T07:35:40.902 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T07:35:40.902 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout disabled 2024-06-06T07:35:40.910 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T07:35:40.910 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout inactive 2024-06-06T07:35:40.910 INFO:teuthology.orchestra.run.smithi082.stdout:firewalld.service is not enabled 2024-06-06T07:35:40.910 INFO:teuthology.orchestra.run.smithi082.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-06-06T07:35:40.910 INFO:teuthology.orchestra.run.smithi082.stdout:Waiting for mgr to start... 2024-06-06T07:35:40.910 INFO:teuthology.orchestra.run.smithi082.stdout:Waiting for mgr... 2024-06-06T07:35:41.500 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:41 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1069040147' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-06T07:35:41.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:41.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout { 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "fsid": "355cdb60-23d7-11ef-bc9b-c7b262605968", 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "health": { 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 0 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout ], 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "smithi082" 2024-06-06T07:35:41.966 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout ], 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-06T07:35:41.967 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-06T07:35:41.968 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-06T07:35:41.969 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "restful" 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout ], 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "modified": "2024-06-06T07:35:28.757294+0000", 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout } 2024-06-06T07:35:41.970 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.302+0000 7f3ddeea9640 1 Processor -- start 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.302+0000 7f3ddeea9640 1 -- start start 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.302+0000 7f3ddeea9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd8106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.302+0000 7f3ddeea9640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd8106630 con 0x7f3dd8105c40 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.303+0000 7f3ddcc45640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd8106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.303+0000 7f3ddcc45640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd8106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42772/0 (socket says 172.21.15.82:42772) 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.303+0000 7f3ddcc45640 1 -- 172.21.15.82:0/3486632585 learned_addr learned my addr 172.21.15.82:0/3486632585 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.303+0000 7f3ddcc45640 1 -- 172.21.15.82:0/3486632585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd8106e40 con 0x7f3dd8105c40 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3ddcc45640 1 --2- 172.21.15.82:0/3486632585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd8106060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009820 tx=0x7f3dcc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f230053494a2a69 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3dd77fe640 1 -- 172.21.15.82:0/3486632585 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dcc02ad40 con 0x7f3dd8105c40 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3dd77fe640 1 -- 172.21.15.82:0/3486632585 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3dcc0328c0 con 0x7f3dd8105c40 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3dd77fe640 1 -- 172.21.15.82:0/3486632585 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dcc0307f0 con 0x7f3dd8105c40 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3ddeea9640 1 -- 172.21.15.82:0/3486632585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 msgr2=0x7f3dd8106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:41.971 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3ddeea9640 1 --2- 172.21.15.82:0/3486632585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd8106060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009820 tx=0x7f3dcc02a270 comp rx=0 tx=0).stop 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3ddeea9640 1 -- 172.21.15.82:0/3486632585 shutdown_connections 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3ddeea9640 1 --2- 172.21.15.82:0/3486632585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd8106060 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3ddeea9640 1 -- 172.21.15.82:0/3486632585 >> 172.21.15.82:0/3486632585 conn(0x7f3dd81011a0 msgr2=0x7f3dd8103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3ddeea9640 1 -- 172.21.15.82:0/3486632585 shutdown_connections 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.304+0000 7f3ddeea9640 1 -- 172.21.15.82:0/3486632585 wait complete. 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.305+0000 7f3ddeea9640 1 Processor -- start 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.305+0000 7f3ddeea9640 1 -- start start 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.305+0000 7f3ddeea9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd807bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.305+0000 7f3ddeea9640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd807bff0 con 0x7f3dd8105c40 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.305+0000 7f3ddcc45640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd807bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.305+0000 7f3ddcc45640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd807bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42788/0 (socket says 172.21.15.82:42788) 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.305+0000 7f3ddcc45640 1 -- 172.21.15.82:0/1069040147 learned_addr learned my addr 172.21.15.82:0/1069040147 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.306+0000 7f3ddcc45640 1 -- 172.21.15.82:0/1069040147 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc0094d0 con 0x7f3dd8105c40 2024-06-06T07:35:41.972 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.306+0000 7f3ddcc45640 1 --2- 172.21.15.82:0/1069040147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd807bab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc030d30 tx=0x7f3dcc030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.306+0000 7f3dd5ffb640 1 -- 172.21.15.82:0/1069040147 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dcc032c40 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.306+0000 7f3dd5ffb640 1 -- 172.21.15.82:0/1069040147 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3dcc032da0 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.306+0000 7f3ddeea9640 1 -- 172.21.15.82:0/1069040147 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd807a230 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.306+0000 7f3dd5ffb640 1 -- 172.21.15.82:0/1069040147 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dcc03c6a0 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.306+0000 7f3ddeea9640 1 -- 172.21.15.82:0/1069040147 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd807a650 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.306+0000 7f3dd5ffb640 1 -- 172.21.15.82:0/1069040147 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f3dcc03cc10 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.306+0000 7f3dd5ffb640 1 -- 172.21.15.82:0/1069040147 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3dcc0463c0 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.307+0000 7f3ddeea9640 1 -- 172.21.15.82:0/1069040147 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd8061ae0 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.309+0000 7f3dd5ffb640 1 -- 172.21.15.82:0/1069040147 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f3dcc037070 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.372+0000 7f3ddeea9640 1 -- 172.21.15.82:0/1069040147 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3dd810a070 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.372+0000 7f3dd5ffb640 1 -- 172.21.15.82:0/1069040147 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f3dcc03da90 con 0x7f3dd8105c40 2024-06-06T07:35:41.973 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.373+0000 7f3ddeea9640 1 -- 172.21.15.82:0/1069040147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 msgr2=0x7f3dd807bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:41.974 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.373+0000 7f3ddeea9640 1 --2- 172.21.15.82:0/1069040147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd807bab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc030d30 tx=0x7f3dcc030d60 comp rx=0 tx=0).stop 2024-06-06T07:35:41.974 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.373+0000 7f3ddeea9640 1 -- 172.21.15.82:0/1069040147 shutdown_connections 2024-06-06T07:35:41.974 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.373+0000 7f3ddeea9640 1 --2- 172.21.15.82:0/1069040147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3dd8105c40 0x7f3dd807bab0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:41.974 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.373+0000 7f3ddeea9640 1 -- 172.21.15.82:0/1069040147 >> 172.21.15.82:0/1069040147 conn(0x7f3dd81011a0 msgr2=0x7f3dd8101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:41.974 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.373+0000 7f3ddeea9640 1 -- 172.21.15.82:0/1069040147 shutdown_connections 2024-06-06T07:35:41.974 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:41.373+0000 7f3ddeea9640 1 -- 172.21.15.82:0/1069040147 wait complete. 2024-06-06T07:35:41.974 INFO:teuthology.orchestra.run.smithi082.stdout:mgr not available, waiting (1/15)... 2024-06-06T07:35:46.211 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: Activating manager daemon smithi082.kocfig 2024-06-06T07:35:46.211 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: mgrmap e2: smithi082.kocfig(active, starting, since 0.00322316s) 2024-06-06T07:35:46.211 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T07:35:46.211 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T07:35:46.211 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T07:35:46.211 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:35:46.211 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr metadata", "who": "smithi082.kocfig", "id": "smithi082.kocfig"}]: dispatch 2024-06-06T07:35:46.212 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: Manager daemon smithi082.kocfig is now available 2024-06-06T07:35:46.212 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi082.kocfig/mirror_snapshot_schedule"}]: dispatch 2024-06-06T07:35:46.212 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi082.kocfig/trash_purge_schedule"}]: dispatch 2024-06-06T07:35:46.212 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:46.212 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:46.212 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:45 smithi082 ceph-mon[29354]: from='mgr.14100 172.21.15.82:0/2284163052' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:47.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:48.200 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout { 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "fsid": "355cdb60-23d7-11ef-bc9b-c7b262605968", 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "health": { 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 0 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout ], 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-06T07:35:48.201 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "smithi082" 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout ], 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T07:35:48.202 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-06T07:35:48.203 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:48.204 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "restful" 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout ], 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "modified": "2024-06-06T07:35:28.757294+0000", 2024-06-06T07:35:48.205 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout }, 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout } 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.152+0000 7f039e317640 1 Processor -- start 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.152+0000 7f039e317640 1 -- start start 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.152+0000 7f039e317640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398108b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.152+0000 7f039e317640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0398079e00 con 0x7f0398106730 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.153+0000 7f0397fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398108b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.153+0000 7f0397fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398108b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52724/0 (socket says 172.21.15.82:52724) 2024-06-06T07:35:48.206 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.153+0000 7f0397fff640 1 -- 172.21.15.82:0/1375619903 learned_addr learned my addr 172.21.15.82:0/1375619903 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.153+0000 7f0397fff640 1 -- 172.21.15.82:0/1375619903 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0398109060 con 0x7f0398106730 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.154+0000 7f0397fff640 1 --2- 172.21.15.82:0/1375619903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398108b20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0388009820 tx=0x7f038802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80437fdd1df35bfe server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.154+0000 7f0396ffd640 1 -- 172.21.15.82:0/1375619903 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f038802ad40 con 0x7f0398106730 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.154+0000 7f0396ffd640 1 -- 172.21.15.82:0/1375619903 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f03880328c0 con 0x7f0398106730 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.154+0000 7f0396ffd640 1 -- 172.21.15.82:0/1375619903 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03880307f0 con 0x7f0398106730 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.155+0000 7f039e317640 1 -- 172.21.15.82:0/1375619903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 msgr2=0x7f0398108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.155+0000 7f039e317640 1 --2- 172.21.15.82:0/1375619903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398108b20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0388009820 tx=0x7f038802a270 comp rx=0 tx=0).stop 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.155+0000 7f039e317640 1 -- 172.21.15.82:0/1375619903 shutdown_connections 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.155+0000 7f039e317640 1 --2- 172.21.15.82:0/1375619903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398108b20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.155+0000 7f039e317640 1 -- 172.21.15.82:0/1375619903 >> 172.21.15.82:0/1375619903 conn(0x7f03981000e0 msgr2=0x7f0398102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.155+0000 7f039e317640 1 -- 172.21.15.82:0/1375619903 shutdown_connections 2024-06-06T07:35:48.207 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.155+0000 7f039e317640 1 -- 172.21.15.82:0/1375619903 wait complete. 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.155+0000 7f039e317640 1 Processor -- start 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.155+0000 7f039e317640 1 -- start start 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.156+0000 7f039e317640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.156+0000 7f039e317640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0398199d80 con 0x7f0398106730 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.156+0000 7f0397fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.156+0000 7f0397fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52730/0 (socket says 172.21.15.82:52730) 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.156+0000 7f0397fff640 1 -- 172.21.15.82:0/3725837105 learned_addr learned my addr 172.21.15.82:0/3725837105 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.156+0000 7f0397fff640 1 -- 172.21.15.82:0/3725837105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880094d0 con 0x7f0398106730 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.156+0000 7f0397fff640 1 --2- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398199840 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0388009940 tx=0x7f0388032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.157+0000 7f03957fa640 1 -- 172.21.15.82:0/3725837105 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0388032df0 con 0x7f0398106730 2024-06-06T07:35:48.208 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.157+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0398199f80 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.157+0000 7f03957fa640 1 -- 172.21.15.82:0/3725837105 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0388030cc0 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.157+0000 7f03957fa640 1 -- 172.21.15.82:0/3725837105 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f038803c6b0 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.157+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039819a3a0 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.157+0000 7f03957fa640 1 -- 172.21.15.82:0/3725837105 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 3) v1 ==== 44749+0+0 (secure 0 0 0) 0x7f038803c920 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.158+0000 7f03957fa640 1 --2- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f0380037740 0x7f0380039c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.158+0000 7f03957fa640 1 -- 172.21.15.82:0/3725837105 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f038806a5b0 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.158+0000 7f03977fe640 1 --2- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f0380037740 0x7f0380039c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.158+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0398061ae0 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.158+0000 7f03977fe640 1 --2- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f0380037740 0x7f0380039c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f038c0098c0 tx=0x7f038c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.163+0000 7f03957fa640 1 -- 172.21.15.82:0/3725837105 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f03880313a0 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.398+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0398109f70 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.398+0000 7f03957fa640 1 -- 172.21.15.82:0/3725837105 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1245 (secure 0 0 0) 0x7f038803aeb0 con 0x7f0398106730 2024-06-06T07:35:48.209 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f0380037740 msgr2=0x7f0380039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 --2- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f0380037740 0x7f0380039c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f038c0098c0 tx=0x7f038c006db0 comp rx=0 tx=0).stop 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 msgr2=0x7f0398199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 --2- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398199840 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0388009940 tx=0x7f0388032c60 comp rx=0 tx=0).stop 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 shutdown_connections 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 --2- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f0380037740 0x7f0380039c00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 --2- 172.21.15.82:0/3725837105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0398106730 0x7f0398199840 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 >> 172.21.15.82:0/3725837105 conn(0x7f03981000e0 msgr2=0x7f0398100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 shutdown_connections 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:47.400+0000 7f039e317640 1 -- 172.21.15.82:0/3725837105 wait complete. 2024-06-06T07:35:48.210 INFO:teuthology.orchestra.run.smithi082.stdout:mgr is available 2024-06-06T07:35:48.211 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:47 smithi082 ceph-mon[29354]: mgrmap e3: smithi082.kocfig(active, since 1.00641s) 2024-06-06T07:35:48.211 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:47 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3725837105' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-06T07:35:49.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:48 smithi082 ceph-mon[29354]: mgrmap e4: smithi082.kocfig(active, since 2s) 2024-06-06T07:35:49.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:48 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3034528902' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-06T07:35:49.143 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:49.143 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout [global] 2024-06-06T07:35:49.143 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout fsid = 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:35:49.143 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-06T07:35:49.143 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-06T07:35:49.143 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout [osd] 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.326+0000 7f3961865640 1 Processor -- start 2024-06-06T07:35:49.144 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.326+0000 7f3961865640 1 -- start start 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.326+0000 7f3961865640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c108a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.326+0000 7f3961865640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c079e40 con 0x7f395c106680 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.326+0000 7f395affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.326+0000 7f395affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52738/0 (socket says 172.21.15.82:52738) 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.326+0000 7f395affd640 1 -- 172.21.15.82:0/3566595428 learned_addr learned my addr 172.21.15.82:0/3566595428 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.326+0000 7f395affd640 1 -- 172.21.15.82:0/3566595428 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c108fb0 con 0x7f395c106680 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.327+0000 7f395affd640 1 --2- 172.21.15.82:0/3566595428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c108a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f394c009820 tx=0x7f394c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7145c94f76003a server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.327+0000 7f395a7fc640 1 -- 172.21.15.82:0/3566595428 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394c02ad40 con 0x7f395c106680 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.327+0000 7f395a7fc640 1 -- 172.21.15.82:0/3566595428 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f394c0328c0 con 0x7f395c106680 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.327+0000 7f395a7fc640 1 -- 172.21.15.82:0/3566595428 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394c0307f0 con 0x7f395c106680 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.328+0000 7f3961865640 1 -- 172.21.15.82:0/3566595428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 msgr2=0x7f395c108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.328+0000 7f3961865640 1 --2- 172.21.15.82:0/3566595428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c108a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f394c009820 tx=0x7f394c02a270 comp rx=0 tx=0).stop 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.328+0000 7f3961865640 1 -- 172.21.15.82:0/3566595428 shutdown_connections 2024-06-06T07:35:49.145 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.328+0000 7f3961865640 1 --2- 172.21.15.82:0/3566595428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c108a70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.328+0000 7f3961865640 1 -- 172.21.15.82:0/3566595428 >> 172.21.15.82:0/3566595428 conn(0x7f395c1000d0 msgr2=0x7f395c1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.328+0000 7f3961865640 1 -- 172.21.15.82:0/3566595428 shutdown_connections 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.328+0000 7f3961865640 1 -- 172.21.15.82:0/3566595428 wait complete. 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f3961865640 1 Processor -- start 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f3961865640 1 -- start start 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f3961865640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f3961865640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c199d80 con 0x7f395c106680 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f395affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f395affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52754/0 (socket says 172.21.15.82:52754) 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f395affd640 1 -- 172.21.15.82:0/3034528902 learned_addr learned my addr 172.21.15.82:0/3034528902 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f395affd640 1 -- 172.21.15.82:0/3034528902 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c0094d0 con 0x7f395c106680 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f395affd640 1 --2- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c199840 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f394c009940 tx=0x7f394c032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f3960863640 1 -- 172.21.15.82:0/3034528902 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394c032df0 con 0x7f395c106680 2024-06-06T07:35:49.146 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f3960863640 1 -- 172.21.15.82:0/3034528902 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f394c030cc0 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395c199f80 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f3960863640 1 -- 172.21.15.82:0/3034528902 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394c03c6b0 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.329+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395c19a3a0 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.330+0000 7f3960863640 1 -- 172.21.15.82:0/3034528902 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 4) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f394c03c970 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.330+0000 7f3960863640 1 --2- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3948037810 0x7f3948039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.330+0000 7f3960863640 1 -- 172.21.15.82:0/3034528902 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f394c06a7d0 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.331+0000 7f3953fff640 1 --2- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3948037810 0x7f3948039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.331+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395c061ae0 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.331+0000 7f3953fff640 1 --2- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3948037810 0x7f3948039cd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f39440098c0 tx=0x7f3944006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.335+0000 7f3960863640 1 -- 172.21.15.82:0/3034528902 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f394c03bda0 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.506+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f395c109d30 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.507+0000 7f3960863640 1 -- 172.21.15.82:0/3034528902 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f394c03b340 con 0x7f395c106680 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3948037810 msgr2=0x7f3948039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 --2- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3948037810 0x7f3948039cd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f39440098c0 tx=0x7f3944006db0 comp rx=0 tx=0).stop 2024-06-06T07:35:49.147 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 msgr2=0x7f395c199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:49.148 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 --2- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c199840 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f394c009940 tx=0x7f394c032c60 comp rx=0 tx=0).stop 2024-06-06T07:35:49.148 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 shutdown_connections 2024-06-06T07:35:49.148 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 --2- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3948037810 0x7f3948039cd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:49.148 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 --2- 172.21.15.82:0/3034528902 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f395c106680 0x7f395c199840 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:49.148 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 >> 172.21.15.82:0/3034528902 conn(0x7f395c1000d0 msgr2=0x7f395c100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:49.148 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 shutdown_connections 2024-06-06T07:35:49.148 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:48.509+0000 7f3961865640 1 -- 172.21.15.82:0/3034528902 wait complete. 2024-06-06T07:35:49.148 INFO:teuthology.orchestra.run.smithi082.stdout:Enabling cephadm module... 2024-06-06T07:35:50.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:49 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1368148643' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-06T07:35:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.491+0000 7fe6917e0640 1 Processor -- start 2024-06-06T07:35:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.491+0000 7fe6917e0640 1 -- start start 2024-06-06T07:35:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.491+0000 7fe6917e0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c07a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.491+0000 7fe6917e0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c07a860 con 0x7fe68c07bde0 2024-06-06T07:35:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.491+0000 7fe68affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c07a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.491+0000 7fe68affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c07a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52766/0 (socket says 172.21.15.82:52766) 2024-06-06T07:35:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.491+0000 7fe68affd640 1 -- 172.21.15.82:0/4003566240 learned_addr learned my addr 172.21.15.82:0/4003566240 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.492+0000 7fe68affd640 1 -- 172.21.15.82:0/4003566240 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c07a9a0 con 0x7fe68c07bde0 2024-06-06T07:35:50.382 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.492+0000 7fe68affd640 1 --2- 172.21.15.82:0/4003566240 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c07a290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009820 tx=0x7fe67c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995b0e256c91b030 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.492+0000 7fe689ffb640 1 -- 172.21.15.82:0/4003566240 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe67c02ad40 con 0x7fe68c07bde0 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.492+0000 7fe689ffb640 1 -- 172.21.15.82:0/4003566240 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe67c0328c0 con 0x7fe68c07bde0 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.492+0000 7fe689ffb640 1 -- 172.21.15.82:0/4003566240 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe67c0307f0 con 0x7fe68c07bde0 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.493+0000 7fe6917e0640 1 -- 172.21.15.82:0/4003566240 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 msgr2=0x7fe68c07a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.493+0000 7fe6917e0640 1 --2- 172.21.15.82:0/4003566240 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c07a290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009820 tx=0x7fe67c02a270 comp rx=0 tx=0).stop 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.493+0000 7fe6917e0640 1 -- 172.21.15.82:0/4003566240 shutdown_connections 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.493+0000 7fe6917e0640 1 --2- 172.21.15.82:0/4003566240 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c07a290 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.493+0000 7fe6917e0640 1 -- 172.21.15.82:0/4003566240 >> 172.21.15.82:0/4003566240 conn(0x7fe68c101270 msgr2=0x7fe68c103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.493+0000 7fe6917e0640 1 -- 172.21.15.82:0/4003566240 shutdown_connections 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.493+0000 7fe6917e0640 1 -- 172.21.15.82:0/4003566240 wait complete. 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.493+0000 7fe6917e0640 1 Processor -- start 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.494+0000 7fe6917e0640 1 -- start start 2024-06-06T07:35:50.383 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.494+0000 7fe6917e0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c19ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.494+0000 7fe6917e0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c19bf50 con 0x7fe68c07bde0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.494+0000 7fe68affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c19ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.494+0000 7fe68affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c19ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52780/0 (socket says 172.21.15.82:52780) 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.494+0000 7fe68affd640 1 -- 172.21.15.82:0/1368148643 learned_addr learned my addr 172.21.15.82:0/1368148643 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.494+0000 7fe68affd640 1 -- 172.21.15.82:0/1368148643 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0094d0 con 0x7fe68c07bde0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.494+0000 7fe68affd640 1 --2- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c19ba10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe67c030d30 tx=0x7fe67c030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.494+0000 7fe673fff640 1 -- 172.21.15.82:0/1368148643 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe67c032c40 con 0x7fe68c07bde0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.495+0000 7fe673fff640 1 -- 172.21.15.82:0/1368148643 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe67c032da0 con 0x7fe68c07bde0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.495+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68c19c150 con 0x7fe68c07bde0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.495+0000 7fe673fff640 1 -- 172.21.15.82:0/1368148643 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe67c03c710 con 0x7fe68c07bde0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.495+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68c19c570 con 0x7fe68c07bde0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.495+0000 7fe673fff640 1 -- 172.21.15.82:0/1368148643 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 4) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fe67c03c870 con 0x7fe68c07bde0 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.496+0000 7fe673fff640 1 --2- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7fe674037860 0x7fe674039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:50.384 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.496+0000 7fe673fff640 1 -- 172.21.15.82:0/1368148643 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe67c06a940 con 0x7fe68c07bde0 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.496+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68c10bf90 con 0x7fe68c07bde0 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.496+0000 7fe68a7fc640 1 --2- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7fe674037860 0x7fe674039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.496+0000 7fe68a7fc640 1 --2- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7fe674037860 0x7fe674039d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe6800098c0 tx=0x7fe680006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.500+0000 7fe673fff640 1 -- 172.21.15.82:0/1368148643 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe67c0313a0 con 0x7fe68c07bde0 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.716+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fe68c19f170 con 0x7fe68c07bde0 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.742+0000 7fe673fff640 1 -- 172.21.15.82:0/1368148643 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 5) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fe67c047980 con 0x7fe68c07bde0 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.743+0000 7fe673fff640 1 -- 172.21.15.82:0/1368148643 <== mon.0 v2:172.21.15.82:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe67c02f2a0 con 0x7fe68c07bde0 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.745+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7fe674037860 msgr2=0x7fe674039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.745+0000 7fe6917e0640 1 --2- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7fe674037860 0x7fe674039d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe6800098c0 tx=0x7fe680006db0 comp rx=0 tx=0).stop 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.745+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 msgr2=0x7fe68c19ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.745+0000 7fe6917e0640 1 --2- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c19ba10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe67c030d30 tx=0x7fe67c030d60 comp rx=0 tx=0).stop 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.745+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 shutdown_connections 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.745+0000 7fe6917e0640 1 --2- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7fe674037860 0x7fe674039d20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:50.385 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.745+0000 7fe6917e0640 1 --2- 172.21.15.82:0/1368148643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe68c07bde0 0x7fe68c19ba10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:50.386 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.745+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 >> 172.21.15.82:0/1368148643 conn(0x7fe68c101270 msgr2=0x7fe68c101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:50.386 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.746+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 shutdown_connections 2024-06-06T07:35:50.386 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:49.746+0000 7fe6917e0640 1 -- 172.21.15.82:0/1368148643 wait complete. 2024-06-06T07:35:50.994 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:50 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1368148643' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-06T07:35:50.994 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:50 smithi082 ceph-mon[29354]: mgrmap e5: smithi082.kocfig(active, since 4s) 2024-06-06T07:35:51.647 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout { 2024-06-06T07:35:51.647 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-06T07:35:51.647 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-06T07:35:51.647 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "active_name": "smithi082.kocfig", 2024-06-06T07:35:51.647 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-06T07:35:51.647 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout } 2024-06-06T07:35:51.647 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.788+0000 7f3a315d9640 1 Processor -- start 2024-06-06T07:35:51.647 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.788+0000 7f3a315d9640 1 -- start start 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.788+0000 7f3a315d9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c108ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.788+0000 7f3a315d9640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c079e40 con 0x7f3a2c1066e0 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.788+0000 7f3a2affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c108ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.788+0000 7f3a2affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c108ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52808/0 (socket says 172.21.15.82:52808) 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.788+0000 7f3a2affd640 1 -- 172.21.15.82:0/2080637853 learned_addr learned my addr 172.21.15.82:0/2080637853 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.789+0000 7f3a2affd640 1 -- 172.21.15.82:0/2080637853 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c109010 con 0x7f3a2c1066e0 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.789+0000 7f3a2affd640 1 --2- 172.21.15.82:0/2080637853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c108ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009820 tx=0x7f3a1c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89a85928f07f21b server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.789+0000 7f3a29ffb640 1 -- 172.21.15.82:0/2080637853 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a1c02ad40 con 0x7f3a2c1066e0 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.789+0000 7f3a29ffb640 1 -- 172.21.15.82:0/2080637853 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3a1c0328c0 con 0x7f3a2c1066e0 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.789+0000 7f3a29ffb640 1 -- 172.21.15.82:0/2080637853 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a1c0307f0 con 0x7f3a2c1066e0 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.790+0000 7f3a315d9640 1 -- 172.21.15.82:0/2080637853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 msgr2=0x7f3a2c108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.790+0000 7f3a315d9640 1 --2- 172.21.15.82:0/2080637853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c108ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009820 tx=0x7f3a1c02a270 comp rx=0 tx=0).stop 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.790+0000 7f3a315d9640 1 -- 172.21.15.82:0/2080637853 shutdown_connections 2024-06-06T07:35:51.648 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.790+0000 7f3a315d9640 1 --2- 172.21.15.82:0/2080637853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c108ad0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.790+0000 7f3a315d9640 1 -- 172.21.15.82:0/2080637853 >> 172.21.15.82:0/2080637853 conn(0x7f3a2c1000d0 msgr2=0x7f3a2c102550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.790+0000 7f3a315d9640 1 -- 172.21.15.82:0/2080637853 shutdown_connections 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.790+0000 7f3a315d9640 1 -- 172.21.15.82:0/2080637853 wait complete. 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.791+0000 7f3a315d9640 1 Processor -- start 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.791+0000 7f3a315d9640 1 -- start start 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.791+0000 7f3a315d9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.791+0000 7f3a315d9640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c199d90 con 0x7f3a2c1066e0 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.791+0000 7f3a2affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.791+0000 7f3a2affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52820/0 (socket says 172.21.15.82:52820) 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.791+0000 7f3a2affd640 1 -- 172.21.15.82:0/2046911303 learned_addr learned my addr 172.21.15.82:0/2046911303 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.791+0000 7f3a2affd640 1 -- 172.21.15.82:0/2046911303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c0094d0 con 0x7f3a2c1066e0 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.792+0000 7f3a2affd640 1 --2- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c199850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c030d30 tx=0x7f3a1c030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:51.649 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.792+0000 7f3a23fff640 1 -- 172.21.15.82:0/2046911303 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a1c032c40 con 0x7f3a2c1066e0 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.792+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2c199f90 con 0x7f3a2c1066e0 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.792+0000 7f3a23fff640 1 -- 172.21.15.82:0/2046911303 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3a1c032da0 con 0x7f3a2c1066e0 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.792+0000 7f3a23fff640 1 -- 172.21.15.82:0/2046911303 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a1c03c710 con 0x7f3a2c1066e0 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.792+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2c19a3b0 con 0x7f3a2c1066e0 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.792+0000 7f3a23fff640 1 -- 172.21.15.82:0/2046911303 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 5) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f3a1c03c980 con 0x7f3a2c1066e0 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.793+0000 7f3a23fff640 1 --2- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3a18037860 0x7f3a18039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.793+0000 7f3a23fff640 1 -- 172.21.15.82:0/2046911303 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3a1c06b660 con 0x7f3a2c1066e0 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.793+0000 7f3a2a7fc640 1 -- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3a18037860 msgr2=0x7f3a18039d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1192692524 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.793+0000 7f3a2a7fc640 1 --2- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3a18037860 0x7f3a18039d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.793+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2c061ae0 con 0x7f3a2c1066e0 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.802+0000 7f3a23fff640 1 -- 172.21.15.82:0/2046911303 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3a1c0313a0 con 0x7f3a2c1066e0 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.994+0000 7f3a2a7fc640 1 -- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3a18037860 msgr2=0x7f3a18039d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1192692524 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:50.994+0000 7f3a2a7fc640 1 --2- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3a18037860 0x7f3a18039d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T07:35:51.650 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.027+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3a2c109fc0 con 0x7f3a2c1066e0 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.027+0000 7f3a23fff640 1 -- 172.21.15.82:0/2046911303 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3a1c03bb30 con 0x7f3a2c1066e0 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.029+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3a18037860 msgr2=0x7f3a18039d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.029+0000 7f3a315d9640 1 --2- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3a18037860 0x7f3a18039d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.029+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 msgr2=0x7f3a2c199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.029+0000 7f3a315d9640 1 --2- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c199850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c030d30 tx=0x7f3a1c030d60 comp rx=0 tx=0).stop 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.029+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 shutdown_connections 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.029+0000 7f3a315d9640 1 --2- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3a18037860 0x7f3a18039d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.029+0000 7f3a315d9640 1 --2- 172.21.15.82:0/2046911303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a2c1066e0 0x7f3a2c199850 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.029+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 >> 172.21.15.82:0/2046911303 conn(0x7f3a2c1000d0 msgr2=0x7f3a2c100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.030+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 shutdown_connections 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:51.030+0000 7f3a315d9640 1 -- 172.21.15.82:0/2046911303 wait complete. 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:Waiting for the mgr to restart... 2024-06-06T07:35:51.651 INFO:teuthology.orchestra.run.smithi082.stdout:Waiting for mgr epoch 5... 2024-06-06T07:35:51.905 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:51 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2046911303' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-06T07:35:54.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: Active manager daemon smithi082.kocfig restarted 2024-06-06T07:35:54.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: Activating manager daemon smithi082.kocfig 2024-06-06T07:35:54.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: osdmap e2: 0 total, 0 up, 0 in 2024-06-06T07:35:54.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: mgrmap e6: smithi082.kocfig(active, starting, since 0.00373972s) 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr metadata", "who": "smithi082.kocfig", "id": "smithi082.kocfig"}]: dispatch 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: Manager daemon smithi082.kocfig is now available 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi082.kocfig/mirror_snapshot_schedule"}]: dispatch 2024-06-06T07:35:54.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:54 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi082.kocfig/trash_purge_schedule"}]: dispatch 2024-06-06T07:35:55.919 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:55 smithi082 ceph-mon[29354]: Found migration_current of "None". Setting to last migration. 2024-06-06T07:35:55.919 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:55 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:55.919 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:55 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:55.919 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:55 smithi082 ceph-mon[29354]: mgrmap e7: smithi082.kocfig(active, since 1.00548s) 2024-06-06T07:35:56.222 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout { 2024-06-06T07:35:56.222 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-06T07:35:56.222 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-06T07:35:56.222 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout } 2024-06-06T07:35:56.222 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.095+0000 7f3b2f8bb640 1 Processor -- start 2024-06-06T07:35:56.222 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.095+0000 7f3b2f8bb640 1 -- start start 2024-06-06T07:35:56.222 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.096+0000 7f3b2f8bb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28104300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:56.222 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.096+0000 7f3b2f8bb640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281048d0 con 0x7f3b28103ee0 2024-06-06T07:35:56.222 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.096+0000 7f3b2d657640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28104300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.096+0000 7f3b2d657640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28104300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52826/0 (socket says 172.21.15.82:52826) 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.096+0000 7f3b2d657640 1 -- 172.21.15.82:0/3648498723 learned_addr learned my addr 172.21.15.82:0/3648498723 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.096+0000 7f3b2d657640 1 -- 172.21.15.82:0/3648498723 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28104a10 con 0x7f3b28103ee0 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.097+0000 7f3b2d657640 1 --2- 172.21.15.82:0/3648498723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28104300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009820 tx=0x7f3b2402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3acbe9df5966e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.097+0000 7f3b1bfff640 1 -- 172.21.15.82:0/3648498723 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b2402ad40 con 0x7f3b28103ee0 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.097+0000 7f3b1bfff640 1 -- 172.21.15.82:0/3648498723 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3b240328c0 con 0x7f3b28103ee0 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.097+0000 7f3b1bfff640 1 -- 172.21.15.82:0/3648498723 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b240307f0 con 0x7f3b28103ee0 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/3648498723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 msgr2=0x7f3b28104300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 --2- 172.21.15.82:0/3648498723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28104300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009820 tx=0x7f3b2402a270 comp rx=0 tx=0).stop 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/3648498723 shutdown_connections 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 --2- 172.21.15.82:0/3648498723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28104300 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:56.223 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/3648498723 >> 172.21.15.82:0/3648498723 conn(0x7f3b280ff860 msgr2=0x7f3b28101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/3648498723 shutdown_connections 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/3648498723 wait complete. 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 Processor -- start 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 -- start start 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.098+0000 7f3b2f8bb640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b28199d60 con 0x7f3b28103ee0 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.099+0000 7f3b2d657640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.099+0000 7f3b2d657640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28199820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52830/0 (socket says 172.21.15.82:52830) 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.099+0000 7f3b2d657640 1 -- 172.21.15.82:0/1833449216 learned_addr learned my addr 172.21.15.82:0/1833449216 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.099+0000 7f3b2d657640 1 -- 172.21.15.82:0/1833449216 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240094d0 con 0x7f3b28103ee0 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.099+0000 7f3b2d657640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28199820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009940 tx=0x7f3b24032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:56.224 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.099+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b24041070 con 0x7f3b28103ee0 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.099+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/1833449216 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b28199f60 con 0x7f3b28103ee0 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.099+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/1833449216 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2819a3e0 con 0x7f3b28103ee0 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.100+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3b24030cc0 con 0x7f3b28103ee0 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.100+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b24046650 con 0x7f3b28103ee0 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.100+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 5) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f3b240467b0 con 0x7f3b28103ee0 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.101+0000 7f3b1a7fc640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 0x7f3b14039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.101+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 --> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b1403a480 con 0x7f3b140378b0 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.101+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3b2406b590 con 0x7f3b28103ee0 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.101+0000 7f3b2ce56640 1 -- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 msgr2=0x7f3b14039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1192692524 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.101+0000 7f3b2ce56640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 0x7f3b14039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.302+0000 7f3b2ce56640 1 -- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 msgr2=0x7f3b14039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1192692524 2024-06-06T07:35:56.225 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.302+0000 7f3b2ce56640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 0x7f3b14039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.703+0000 7f3b2ce56640 1 -- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 msgr2=0x7f3b14039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1192692524 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:52.703+0000 7f3b2ce56640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 0x7f3b14039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:53.504+0000 7f3b2ce56640 1 -- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 msgr2=0x7f3b14039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1192692524 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:53.504+0000 7f3b2ce56640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 0x7f3b14039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:54.662+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mgrmap(e 6) v1 ==== 44633+0+0 (secure 0 0 0) 0x7f3b2404ccf0 con 0x7f3b28103ee0 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:54.662+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 msgr2=0x7f3b14039d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:54.662+0000 7f3b1a7fc640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 0x7f3b14039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.663+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 7) v1 ==== 44760+0+0 (secure 0 0 0) 0x7f3b2403da90 con 0x7f3b28103ee0 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.663+0000 7f3b1a7fc640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3b1403afc0 0x7f3b1403d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.663+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b1403a480 con 0x7f3b1403afc0 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.663+0000 7f3b2ce56640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3b1403afc0 0x7f3b1403d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:56.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.664+0000 7f3b2ce56640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3b1403afc0 0x7f3b1403d3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c003980 tx=0x7f3b1c006e10 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.664+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f3b1403a480 con 0x7f3b1403afc0 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.670+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/1833449216 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f3b0c0025a0 con 0x7f3b1403afc0 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.670+0000 7f3b1a7fc640 1 -- 172.21.15.82:0/1833449216 <== mgr.14116 v2:172.21.15.82:6800/1662212559 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3b0c0025a0 con 0x7f3b1403afc0 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.670+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3b1403afc0 msgr2=0x7f3b1403d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3b1403afc0 0x7f3b1403d3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c003980 tx=0x7f3b1c006e10 comp rx=0 tx=0).stop 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 msgr2=0x7f3b28199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28199820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009940 tx=0x7f3b24032c60 comp rx=0 tx=0).stop 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/1833449216 shutdown_connections 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3b1403afc0 0x7f3b1403d3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:6800/1192692524,v1:172.21.15.82:6801/1192692524] conn(0x7f3b140378b0 0x7f3b14039d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:56.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 --2- 172.21.15.82:0/1833449216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b28103ee0 0x7f3b28199820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:56.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/1833449216 >> 172.21.15.82:0/1833449216 conn(0x7f3b280ff860 msgr2=0x7f3b281002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:56.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/1833449216 shutdown_connections 2024-06-06T07:35:56.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:55.671+0000 7f3b2f8bb640 1 -- 172.21.15.82:0/1833449216 wait complete. 2024-06-06T07:35:56.228 INFO:teuthology.orchestra.run.smithi082.stdout:mgr epoch 5 is available 2024-06-06T07:35:56.228 INFO:teuthology.orchestra.run.smithi082.stdout:Setting orchestrator backend to cephadm... 2024-06-06T07:35:56.824 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:56 smithi082 ceph-mon[29354]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-06T07:35:56.824 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:56 smithi082 ceph-mon[29354]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-06T07:35:56.824 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:56 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:35:57.392 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.626+0000 7f6e36d08640 1 Processor -- start 2024-06-06T07:35:57.392 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.626+0000 7f6e36d08640 1 -- start start 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.626+0000 7f6e36d08640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3007a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.626+0000 7f6e36d08640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3007a860 con 0x7f6e3007be30 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.626+0000 7f6e34aa4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3007a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.626+0000 7f6e34aa4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3007a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57626/0 (socket says 172.21.15.82:57626) 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.626+0000 7f6e34aa4640 1 -- 172.21.15.82:0/1507599799 learned_addr learned my addr 172.21.15.82:0/1507599799 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.626+0000 7f6e34aa4640 1 -- 172.21.15.82:0/1507599799 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3007a9a0 con 0x7f6e3007be30 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.627+0000 7f6e34aa4640 1 --2- 172.21.15.82:0/1507599799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3007a290 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6e24009820 tx=0x7f6e2402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7674ad1f7252eb5b server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.627+0000 7f6e2f7fe640 1 -- 172.21.15.82:0/1507599799 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e2402ad40 con 0x7f6e3007be30 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.627+0000 7f6e2f7fe640 1 -- 172.21.15.82:0/1507599799 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6e240328c0 con 0x7f6e3007be30 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.627+0000 7f6e2f7fe640 1 -- 172.21.15.82:0/1507599799 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e240307f0 con 0x7f6e3007be30 2024-06-06T07:35:57.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.628+0000 7f6e36d08640 1 -- 172.21.15.82:0/1507599799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 msgr2=0x7f6e3007a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.628+0000 7f6e36d08640 1 --2- 172.21.15.82:0/1507599799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3007a290 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6e24009820 tx=0x7f6e2402a270 comp rx=0 tx=0).stop 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.628+0000 7f6e36d08640 1 -- 172.21.15.82:0/1507599799 shutdown_connections 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.628+0000 7f6e36d08640 1 --2- 172.21.15.82:0/1507599799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3007a290 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.628+0000 7f6e36d08640 1 -- 172.21.15.82:0/1507599799 >> 172.21.15.82:0/1507599799 conn(0x7f6e301011c0 msgr2=0x7f6e30103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.628+0000 7f6e36d08640 1 -- 172.21.15.82:0/1507599799 shutdown_connections 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.628+0000 7f6e36d08640 1 -- 172.21.15.82:0/1507599799 wait complete. 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.628+0000 7f6e36d08640 1 Processor -- start 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e36d08640 1 -- start start 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e36d08640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3019ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e36d08640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3019bfb0 con 0x7f6e3007be30 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e34aa4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3019ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e34aa4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3019ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57640/0 (socket says 172.21.15.82:57640) 2024-06-06T07:35:57.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e34aa4640 1 -- 172.21.15.82:0/2810001927 learned_addr learned my addr 172.21.15.82:0/2810001927 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e34aa4640 1 -- 172.21.15.82:0/2810001927 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e240094d0 con 0x7f6e3007be30 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e34aa4640 1 --2- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3019ba70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6e24009940 tx=0x7f6e24032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e2dffb640 1 -- 172.21.15.82:0/2810001927 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e24041070 con 0x7f6e3007be30 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e2dffb640 1 -- 172.21.15.82:0/2810001927 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6e24030cc0 con 0x7f6e3007be30 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3019c1b0 con 0x7f6e3007be30 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.629+0000 7f6e2dffb640 1 -- 172.21.15.82:0/2810001927 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e2403c690 con 0x7f6e3007be30 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.630+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3019c5d0 con 0x7f6e3007be30 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.630+0000 7f6e2dffb640 1 -- 172.21.15.82:0/2810001927 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 7) v1 ==== 44760+0+0 (secure 0 0 0) 0x7f6e2403c7f0 con 0x7f6e3007be30 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.630+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3010bfa0 con 0x7f6e3007be30 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.631+0000 7f6e2dffb640 1 --2- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6e20037790 0x7f6e20039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:57.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.631+0000 7f6e2dffb640 1 -- 172.21.15.82:0/2810001927 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6e2406a3f0 con 0x7f6e3007be30 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.631+0000 7f6e2ffff640 1 --2- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6e20037790 0x7f6e20039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.631+0000 7f6e2ffff640 1 --2- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6e20037790 0x7f6e20039c50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0098c0 tx=0x7f6e1c006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.635+0000 7f6e2dffb640 1 -- 172.21.15.82:0/2810001927 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6e24031580 con 0x7f6e3007be30 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.716+0000 7f6e2dffb640 1 -- 172.21.15.82:0/2810001927 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f6e24046930 con 0x7f6e3007be30 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.823+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e30102340 con 0x7f6e20037790 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.832+0000 7f6e2dffb640 1 -- 172.21.15.82:0/2810001927 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6e30102340 con 0x7f6e20037790 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6e20037790 msgr2=0x7f6e20039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 --2- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6e20037790 0x7f6e20039c50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0098c0 tx=0x7f6e1c006db0 comp rx=0 tx=0).stop 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 msgr2=0x7f6e3019ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 --2- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3019ba70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6e24009940 tx=0x7f6e24032c60 comp rx=0 tx=0).stop 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 shutdown_connections 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 --2- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6e20037790 0x7f6e20039c50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:57.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 --2- 172.21.15.82:0/2810001927 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e3007be30 0x7f6e3019ba70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:57.397 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 >> 172.21.15.82:0/2810001927 conn(0x7f6e301011c0 msgr2=0x7f6e30101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:57.397 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 shutdown_connections 2024-06-06T07:35:57.397 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:56.834+0000 7f6e36d08640 1 -- 172.21.15.82:0/2810001927 wait complete. 2024-06-06T07:35:57.965 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:57 smithi082 ceph-mon[29354]: [06/Jun/2024:07:35:56] ENGINE Bus STARTING 2024-06-06T07:35:57.965 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:57 smithi082 ceph-mon[29354]: [06/Jun/2024:07:35:56] ENGINE Serving on http://172.21.15.82:8765 2024-06-06T07:35:57.965 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:57 smithi082 ceph-mon[29354]: [06/Jun/2024:07:35:56] ENGINE Serving on https://172.21.15.82:7150 2024-06-06T07:35:57.965 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:57 smithi082 ceph-mon[29354]: [06/Jun/2024:07:35:56] ENGINE Bus STARTED 2024-06-06T07:35:57.965 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:57 smithi082 ceph-mon[29354]: [06/Jun/2024:07:35:56] ENGINE Error in HTTPServer.serve 2024-06-06T07:35:57.965 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Traceback (most recent call last): 2024-06-06T07:35:57.965 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-06T07:35:57.965 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self._connections.run(self.expiration_interval) 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self._run(expiration_interval) 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: s = self.context.wrap_socket( 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: return self.sslsocket_class._create( 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-06T07:35:57.966 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self.do_handshake() 2024-06-06T07:35:57.967 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-06T07:35:57.967 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self._sslobj.do_handshake() 2024-06-06T07:35:57.967 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-06T07:35:57.967 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:57 smithi082 ceph-mon[29354]: mgrmap e8: smithi082.kocfig(active, since 2s) 2024-06-06T07:35:57.967 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:57 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:57.967 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:57 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:35:58.531 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-06T07:35:58.531 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.781+0000 7f6bfb15c640 1 Processor -- start 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.781+0000 7f6bfb15c640 1 -- start start 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.781+0000 7f6bfb15c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4108af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.781+0000 7f6bfb15c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf4079dc0 con 0x7f6bf4106700 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.781+0000 7f6bf8ef8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4108af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.781+0000 7f6bf8ef8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4108af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57646/0 (socket says 172.21.15.82:57646) 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.781+0000 7f6bf8ef8640 1 -- 172.21.15.82:0/1913257898 learned_addr learned my addr 172.21.15.82:0/1913257898 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.781+0000 7f6bf8ef8640 1 -- 172.21.15.82:0/1913257898 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf4109030 con 0x7f6bf4106700 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.782+0000 7f6bf8ef8640 1 --2- 172.21.15.82:0/1913257898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4108af0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009820 tx=0x7f6be802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77fe789fe785be6 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.782+0000 7f6bf37fe640 1 -- 172.21.15.82:0/1913257898 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be802ad40 con 0x7f6bf4106700 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.782+0000 7f6bf37fe640 1 -- 172.21.15.82:0/1913257898 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6be80328c0 con 0x7f6bf4106700 2024-06-06T07:35:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.782+0000 7f6bf37fe640 1 -- 172.21.15.82:0/1913257898 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be80307f0 con 0x7f6bf4106700 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 -- 172.21.15.82:0/1913257898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 msgr2=0x7f6bf4108af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 --2- 172.21.15.82:0/1913257898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4108af0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009820 tx=0x7f6be802a270 comp rx=0 tx=0).stop 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 -- 172.21.15.82:0/1913257898 shutdown_connections 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 --2- 172.21.15.82:0/1913257898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4108af0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 -- 172.21.15.82:0/1913257898 >> 172.21.15.82:0/1913257898 conn(0x7f6bf41000d0 msgr2=0x7f6bf41024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 -- 172.21.15.82:0/1913257898 shutdown_connections 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 -- 172.21.15.82:0/1913257898 wait complete. 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 Processor -- start 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 -- start start 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.783+0000 7f6bfb15c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf4199d70 con 0x7f6bf4106700 2024-06-06T07:35:58.533 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bf8ef8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bf8ef8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57658/0 (socket says 172.21.15.82:57658) 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bf8ef8640 1 -- 172.21.15.82:0/2038544755 learned_addr learned my addr 172.21.15.82:0/2038544755 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bf8ef8640 1 -- 172.21.15.82:0/2038544755 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be80094d0 con 0x7f6bf4106700 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bf8ef8640 1 --2- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4199830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6be8030d30 tx=0x7f6be8030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bf1ffb640 1 -- 172.21.15.82:0/2038544755 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be8032c40 con 0x7f6bf4106700 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bf1ffb640 1 -- 172.21.15.82:0/2038544755 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6be8032da0 con 0x7f6bf4106700 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf4199f70 con 0x7f6bf4106700 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bf1ffb640 1 -- 172.21.15.82:0/2038544755 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be803c710 con 0x7f6bf4106700 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.784+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf419a390 con 0x7f6bf4106700 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.785+0000 7f6bf1ffb640 1 -- 172.21.15.82:0/2038544755 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f6be803d480 con 0x7f6bf4106700 2024-06-06T07:35:58.534 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.785+0000 7f6bf1ffb640 1 --2- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6be4037860 0x7f6be4039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.785+0000 7f6bf1ffb640 1 -- 172.21.15.82:0/2038544755 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6be806aa30 con 0x7f6bf4106700 2024-06-06T07:35:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.785+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf4109d00 con 0x7f6bf4106700 2024-06-06T07:35:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.785+0000 7f6bf3fff640 1 --2- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6be4037860 0x7f6be4039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.786+0000 7f6bf3fff640 1 --2- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6be4037860 0x7f6be4039d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6be00098c0 tx=0x7f6be0006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.790+0000 7f6bf1ffb640 1 -- 172.21.15.82:0/2038544755 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6be8047460 con 0x7f6bf4106700 2024-06-06T07:35:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.964+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bf4101160 con 0x7f6be4037860 2024-06-06T07:35:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.964+0000 7f6bf1ffb640 1 -- 172.21.15.82:0/2038544755 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6bf4101160 con 0x7f6be4037860 2024-06-06T07:35:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.966+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6be4037860 msgr2=0x7f6be4039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:58.535 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.966+0000 7f6bfb15c640 1 --2- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6be4037860 0x7f6be4039d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6be00098c0 tx=0x7f6be0006db0 comp rx=0 tx=0).stop 2024-06-06T07:35:58.536 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.966+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 msgr2=0x7f6bf4199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:58.536 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.966+0000 7f6bfb15c640 1 --2- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4199830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6be8030d30 tx=0x7f6be8030d60 comp rx=0 tx=0).stop 2024-06-06T07:35:58.536 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.966+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 shutdown_connections 2024-06-06T07:35:58.536 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.966+0000 7f6bfb15c640 1 --2- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f6be4037860 0x7f6be4039d20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:58.536 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.966+0000 7f6bfb15c640 1 --2- 172.21.15.82:0/2038544755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bf4106700 0x7f6bf4199830 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:58.536 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.966+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 >> 172.21.15.82:0/2038544755 conn(0x7f6bf41000d0 msgr2=0x7f6bf4100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:58.536 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.967+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 shutdown_connections 2024-06-06T07:35:58.536 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:57.967+0000 7f6bfb15c640 1 -- 172.21.15.82:0/2038544755 wait complete. 2024-06-06T07:35:58.536 INFO:teuthology.orchestra.run.smithi082.stdout:Generating ssh key... 2024-06-06T07:35:58.940 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:58 smithi082 ceph-mon[29354]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:35:59.846 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:59 smithi082 ceph-mon[29354]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:35:59.846 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:59 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:59.846 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:35:59 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:35:59.853 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.939+0000 7f385e56a640 1 Processor -- start 2024-06-06T07:35:59.853 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.939+0000 7f385e56a640 1 -- start start 2024-06-06T07:35:59.853 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.939+0000 7f385e56a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385807a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:59.853 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.939+0000 7f385e56a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385807a8a0 con 0x7f385807be70 2024-06-06T07:35:59.853 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.939+0000 7f3857fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385807a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:59.853 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.939+0000 7f3857fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385807a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57660/0 (socket says 172.21.15.82:57660) 2024-06-06T07:35:59.853 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.939+0000 7f3857fff640 1 -- 172.21.15.82:0/3744065812 learned_addr learned my addr 172.21.15.82:0/3744065812 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.940+0000 7f3857fff640 1 -- 172.21.15.82:0/3744065812 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385807a9e0 con 0x7f385807be70 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.940+0000 7f3857fff640 1 --2- 172.21.15.82:0/3744065812 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385807a2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3848009820 tx=0x7f384802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea15a6a13e8a4fc9 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.940+0000 7f3856ffd640 1 -- 172.21.15.82:0/3744065812 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f384802ad40 con 0x7f385807be70 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.941+0000 7f3856ffd640 1 -- 172.21.15.82:0/3744065812 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f38480328c0 con 0x7f385807be70 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.941+0000 7f3856ffd640 1 -- 172.21.15.82:0/3744065812 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38480307f0 con 0x7f385807be70 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.941+0000 7f385e56a640 1 -- 172.21.15.82:0/3744065812 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 msgr2=0x7f385807a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.941+0000 7f385e56a640 1 --2- 172.21.15.82:0/3744065812 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385807a2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3848009820 tx=0x7f384802a270 comp rx=0 tx=0).stop 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.941+0000 7f385e56a640 1 -- 172.21.15.82:0/3744065812 shutdown_connections 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.941+0000 7f385e56a640 1 --2- 172.21.15.82:0/3744065812 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385807a2d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.941+0000 7f385e56a640 1 -- 172.21.15.82:0/3744065812 >> 172.21.15.82:0/3744065812 conn(0x7f38581011c0 msgr2=0x7f3858103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.941+0000 7f385e56a640 1 -- 172.21.15.82:0/3744065812 shutdown_connections 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.941+0000 7f385e56a640 1 -- 172.21.15.82:0/3744065812 wait complete. 2024-06-06T07:35:59.854 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.942+0000 7f385e56a640 1 Processor -- start 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.942+0000 7f385e56a640 1 -- start start 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.942+0000 7f385e56a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.942+0000 7f385e56a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385819c000 con 0x7f385807be70 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.942+0000 7f3857fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.942+0000 7f3857fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385819bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57664/0 (socket says 172.21.15.82:57664) 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.942+0000 7f3857fff640 1 -- 172.21.15.82:0/4045881008 learned_addr learned my addr 172.21.15.82:0/4045881008 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.942+0000 7f3857fff640 1 -- 172.21.15.82:0/4045881008 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38480094d0 con 0x7f385807be70 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.943+0000 7f3857fff640 1 --2- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385819bac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3848009940 tx=0x7f3848032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.943+0000 7f38557fa640 1 -- 172.21.15.82:0/4045881008 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3848041070 con 0x7f385807be70 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.943+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385819c200 con 0x7f385807be70 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.943+0000 7f38557fa640 1 -- 172.21.15.82:0/4045881008 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3848030cc0 con 0x7f385807be70 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.943+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385819c620 con 0x7f385807be70 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.943+0000 7f38557fa640 1 -- 172.21.15.82:0/4045881008 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f384803c690 con 0x7f385807be70 2024-06-06T07:35:59.855 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.944+0000 7f38557fa640 1 -- 172.21.15.82:0/4045881008 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f384803c7f0 con 0x7f385807be70 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.944+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385810bf80 con 0x7f385807be70 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.944+0000 7f38557fa640 1 --2- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3840037860 0x7f3840039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.944+0000 7f38557fa640 1 -- 172.21.15.82:0/4045881008 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f384806a540 con 0x7f385807be70 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.944+0000 7f38577fe640 1 --2- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3840037860 0x7f3840039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.944+0000 7f38577fe640 1 --2- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3840037860 0x7f3840039d20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f384c0098c0 tx=0x7f384c006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:58.948+0000 7f38557fa640 1 -- 172.21.15.82:0/4045881008 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3848031dd0 con 0x7f385807be70 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.122+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f38581023a0 con 0x7f3840037860 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.288+0000 7f38557fa640 1 -- 172.21.15.82:0/4045881008 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f38581023a0 con 0x7f3840037860 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3840037860 msgr2=0x7f3840039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 --2- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3840037860 0x7f3840039d20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f384c0098c0 tx=0x7f384c006db0 comp rx=0 tx=0).stop 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 msgr2=0x7f385819bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 --2- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385819bac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3848009940 tx=0x7f3848032c60 comp rx=0 tx=0).stop 2024-06-06T07:35:59.856 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 shutdown_connections 2024-06-06T07:35:59.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 --2- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f3840037860 0x7f3840039d20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:59.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 --2- 172.21.15.82:0/4045881008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f385807be70 0x7f385819bac0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:35:59.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 >> 172.21.15.82:0/4045881008 conn(0x7f38581011c0 msgr2=0x7f3858101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:35:59.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 shutdown_connections 2024-06-06T07:35:59.857 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:35:59.290+0000 7f385e56a640 1 -- 172.21.15.82:0/4045881008 wait complete. 2024-06-06T07:36:00.985 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:00 smithi082 ceph-mon[29354]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:00.986 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:00 smithi082 ceph-mon[29354]: Generating ssh key... 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8eK7O538orpl7l6DfGO3buZn8qysEtd8J1eG3M4MyBrn3DS7XQH6JfixkBIyFpJrei8qD+DhfFk02A2C7X1pCCHJWx32s/eCPH4r6e3rZrjNGhuoHpM4H6LrQ9TS3SUV28xSfy2+xYZhquiqLrOphMv7QT0Fca+1I10fSqQcffflJ8Govn5dbIoZ51z1AtLAlGF0KqCV6io1uk1+CHo5Hua1jAdxqI+f4k0LSCNR1es2rkP+XsUXJ66UZkdzpRe0d7wC7A7rX0xKA7ncZjmdtyJVYqZZAWHGhf+zBCyq9gnfv71FCzCt8ZYnkpCpoR3zO53e1EMP+M/DCZTXVbpkpg/EAhng++M45lH2s8yP8MgR1BFk/sgH27S1xN1jBRfnjUhkW2omtNJW7Xjrh7JgsW6hiHLY4Gy5M7u3qwURk8VzdVPq5wE/ve1WKDiPMpuMHuw1RIQaY4JE65TiFFI+gnV3T91s8r6f66H8saR1mXfDGbxoHJgqmEMX8SGCUkr0= ceph-355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.230+0000 7f5656aa3640 1 Processor -- start 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.230+0000 7f5656aa3640 1 -- start start 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.230+0000 7f5656aa3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f5650106080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.230+0000 7f5656aa3640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5650106650 con 0x7f5650105c60 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.230+0000 7f565483f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f5650106080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.230+0000 7f565483f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f5650106080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57674/0 (socket says 172.21.15.82:57674) 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.230+0000 7f565483f640 1 -- 172.21.15.82:0/2101943548 learned_addr learned my addr 172.21.15.82:0/2101943548 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.231+0000 7f565483f640 1 -- 172.21.15.82:0/2101943548 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5650106e60 con 0x7f5650105c60 2024-06-06T07:36:00.993 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.231+0000 7f565483f640 1 --2- 172.21.15.82:0/2101943548 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f5650106080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5644009820 tx=0x7f564402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88855849b2f56de server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.231+0000 7f564f7fe640 1 -- 172.21.15.82:0/2101943548 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564402ad40 con 0x7f5650105c60 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.232+0000 7f564f7fe640 1 -- 172.21.15.82:0/2101943548 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f56440328c0 con 0x7f5650105c60 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.232+0000 7f564f7fe640 1 -- 172.21.15.82:0/2101943548 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56440307f0 con 0x7f5650105c60 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.232+0000 7f5656aa3640 1 -- 172.21.15.82:0/2101943548 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 msgr2=0x7f5650106080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.232+0000 7f5656aa3640 1 --2- 172.21.15.82:0/2101943548 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f5650106080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5644009820 tx=0x7f564402a270 comp rx=0 tx=0).stop 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.232+0000 7f5656aa3640 1 -- 172.21.15.82:0/2101943548 shutdown_connections 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.232+0000 7f5656aa3640 1 --2- 172.21.15.82:0/2101943548 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f5650106080 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.232+0000 7f5656aa3640 1 -- 172.21.15.82:0/2101943548 >> 172.21.15.82:0/2101943548 conn(0x7f56501011c0 msgr2=0x7f5650103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.232+0000 7f5656aa3640 1 -- 172.21.15.82:0/2101943548 shutdown_connections 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.232+0000 7f5656aa3640 1 -- 172.21.15.82:0/2101943548 wait complete. 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.233+0000 7f5656aa3640 1 Processor -- start 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.233+0000 7f5656aa3640 1 -- start start 2024-06-06T07:36:00.994 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.233+0000 7f5656aa3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f565007bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.233+0000 7f5656aa3640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565007bff0 con 0x7f5650105c60 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.233+0000 7f565483f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f565007bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.233+0000 7f565483f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f565007bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57678/0 (socket says 172.21.15.82:57678) 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.233+0000 7f565483f640 1 -- 172.21.15.82:0/3195952731 learned_addr learned my addr 172.21.15.82:0/3195952731 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.233+0000 7f565483f640 1 -- 172.21.15.82:0/3195952731 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56440094d0 con 0x7f5650105c60 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.233+0000 7f565483f640 1 --2- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f565007bab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5644030d30 tx=0x7f5644030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.234+0000 7f564dffb640 1 -- 172.21.15.82:0/3195952731 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5644032c40 con 0x7f5650105c60 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.234+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565007a230 con 0x7f5650105c60 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.234+0000 7f564dffb640 1 -- 172.21.15.82:0/3195952731 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5644032da0 con 0x7f5650105c60 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.234+0000 7f564dffb640 1 -- 172.21.15.82:0/3195952731 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564403c6a0 con 0x7f5650105c60 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.234+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565007a650 con 0x7f5650105c60 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.235+0000 7f564dffb640 1 -- 172.21.15.82:0/3195952731 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f564403c800 con 0x7f5650105c60 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.235+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5650061ae0 con 0x7f5650105c60 2024-06-06T07:36:00.995 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.235+0000 7f564dffb640 1 --2- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f56400378b0 0x7f5640039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.235+0000 7f564dffb640 1 -- 172.21.15.82:0/3195952731 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f564406aac0 con 0x7f5650105c60 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.235+0000 7f564ffff640 1 --2- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f56400378b0 0x7f5640039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.236+0000 7f564ffff640 1 --2- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f56400378b0 0x7f5640039d70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f563c0098c0 tx=0x7f563c006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.240+0000 7f564dffb640 1 -- 172.21.15.82:0/3195952731 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f56440313a0 con 0x7f5650105c60 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.411+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f565010a060 con 0x7f56400378b0 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.412+0000 7f564dffb640 1 -- 172.21.15.82:0/3195952731 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f565010a060 con 0x7f56400378b0 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f56400378b0 msgr2=0x7f5640039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 --2- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f56400378b0 0x7f5640039d70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f563c0098c0 tx=0x7f563c006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 msgr2=0x7f565007bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 --2- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f565007bab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5644030d30 tx=0x7f5644030d60 comp rx=0 tx=0).stop 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 shutdown_connections 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 --2- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f56400378b0 0x7f5640039d70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 --2- 172.21.15.82:0/3195952731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5650105c60 0x7f565007bab0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:00.996 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 >> 172.21.15.82:0/3195952731 conn(0x7f56501011c0 msgr2=0x7f5650101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:00.997 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 shutdown_connections 2024-06-06T07:36:00.997 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:00.414+0000 7f5656aa3640 1 -- 172.21.15.82:0/3195952731 wait complete. 2024-06-06T07:36:00.997 INFO:teuthology.orchestra.run.smithi082.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-06T07:36:00.997 INFO:teuthology.orchestra.run.smithi082.stdout:Adding key to root@localhost authorized_keys... 2024-06-06T07:36:00.997 INFO:teuthology.orchestra.run.smithi082.stdout:Adding host smithi082... 2024-06-06T07:36:01.857 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:01 smithi082 ceph-mon[29354]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:03.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:02 smithi082 ceph-mon[29354]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "addr": "172.21.15.82", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:03.975 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:03 smithi082 ceph-mon[29354]: Deploying cephadm binary to smithi082 2024-06-06T07:36:04.843 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Added host 'smithi082' with addr '172.21.15.82' 2024-06-06T07:36:04.843 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.408+0000 7f2a1d763640 1 Processor -- start 2024-06-06T07:36:04.843 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.408+0000 7f2a1d763640 1 -- start start 2024-06-06T07:36:04.843 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.408+0000 7f2a1d763640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18107650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:04.843 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.408+0000 7f2a1d763640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a18107c20 con 0x7f2a18107230 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.409+0000 7f2a16ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18107650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.409+0000 7f2a16ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18107650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57688/0 (socket says 172.21.15.82:57688) 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.409+0000 7f2a16ffd640 1 -- 172.21.15.82:0/2185510112 learned_addr learned my addr 172.21.15.82:0/2185510112 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.409+0000 7f2a16ffd640 1 -- 172.21.15.82:0/2185510112 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a18107d60 con 0x7f2a18107230 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.409+0000 7f2a16ffd640 1 --2- 172.21.15.82:0/2185510112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18107650 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2a08009e80 tx=0x7f2a0802a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb7aa192726313 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a15ffb640 1 -- 172.21.15.82:0/2185510112 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a08031400 con 0x7f2a18107230 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a15ffb640 1 -- 172.21.15.82:0/2185510112 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2a0802d050 con 0x7f2a18107230 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a15ffb640 1 -- 172.21.15.82:0/2185510112 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a08007910 con 0x7f2a18107230 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a1d763640 1 -- 172.21.15.82:0/2185510112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 msgr2=0x7f2a18107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a1d763640 1 --2- 172.21.15.82:0/2185510112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18107650 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2a08009e80 tx=0x7f2a0802a7a0 comp rx=0 tx=0).stop 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a1d763640 1 -- 172.21.15.82:0/2185510112 shutdown_connections 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a1d763640 1 --2- 172.21.15.82:0/2185510112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18107650 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:04.844 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a1d763640 1 -- 172.21.15.82:0/2185510112 >> 172.21.15.82:0/2185510112 conn(0x7f2a180ff870 msgr2=0x7f2a18101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a1d763640 1 -- 172.21.15.82:0/2185510112 shutdown_connections 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.410+0000 7f2a1d763640 1 -- 172.21.15.82:0/2185510112 wait complete. 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.411+0000 7f2a1d763640 1 Processor -- start 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.411+0000 7f2a1d763640 1 -- start start 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.411+0000 7f2a1d763640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.411+0000 7f2a1d763640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a18106d40 con 0x7f2a18107230 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.411+0000 7f2a16ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.411+0000 7f2a16ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57698/0 (socket says 172.21.15.82:57698) 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.411+0000 7f2a16ffd640 1 -- 172.21.15.82:0/1679783052 learned_addr learned my addr 172.21.15.82:0/1679783052 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.412+0000 7f2a16ffd640 1 -- 172.21.15.82:0/1679783052 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a08009b30 con 0x7f2a18107230 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.412+0000 7f2a16ffd640 1 --2- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18106800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2a08031630 tx=0x7f2a08031660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.412+0000 7f2a0ffff640 1 -- 172.21.15.82:0/1679783052 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a08031c00 con 0x7f2a18107230 2024-06-06T07:36:04.845 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.412+0000 7f2a0ffff640 1 -- 172.21.15.82:0/1679783052 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2a08031d60 con 0x7f2a18107230 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.412+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a18104f80 con 0x7f2a18107230 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.412+0000 7f2a0ffff640 1 -- 172.21.15.82:0/1679783052 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a0803a7a0 con 0x7f2a18107230 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.412+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a18105480 con 0x7f2a18107230 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.413+0000 7f2a0ffff640 1 -- 172.21.15.82:0/1679783052 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f2a08038040 con 0x7f2a18107230 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.413+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a18061ae0 con 0x7f2a18107230 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.413+0000 7f2a0ffff640 1 --2- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f2a040378b0 0x7f2a04039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.413+0000 7f2a0ffff640 1 -- 172.21.15.82:0/1679783052 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2a0806a9f0 con 0x7f2a18107230 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.413+0000 7f2a167fc640 1 --2- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f2a040378b0 0x7f2a04039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.414+0000 7f2a167fc640 1 --2- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f2a040378b0 0x7f2a04039d70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2a000098c0 tx=0x7f2a00006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.418+0000 7f2a0ffff640 1 -- 172.21.15.82:0/1679783052 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2a08042310 con 0x7f2a18107230 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:01.594+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi082", "addr": "172.21.15.82", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a18105ab0 con 0x7f2a040378b0 2024-06-06T07:36:04.846 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.265+0000 7f2a0ffff640 1 -- 172.21.15.82:0/1679783052 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f2a18105ab0 con 0x7f2a040378b0 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f2a040378b0 msgr2=0x7f2a04039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 --2- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f2a040378b0 0x7f2a04039d70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2a000098c0 tx=0x7f2a00006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 msgr2=0x7f2a18106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 --2- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18106800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2a08031630 tx=0x7f2a08031660 comp rx=0 tx=0).stop 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 shutdown_connections 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 --2- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f2a040378b0 0x7f2a04039d70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 --2- 172.21.15.82:0/1679783052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a18107230 0x7f2a18106800 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 >> 172.21.15.82:0/1679783052 conn(0x7f2a180ff870 msgr2=0x7f2a18100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 shutdown_connections 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:04.268+0000 7f2a1d763640 1 -- 172.21.15.82:0/1679783052 wait complete. 2024-06-06T07:36:04.847 INFO:teuthology.orchestra.run.smithi082.stdout:Deploying mon service with default placement... 2024-06-06T07:36:05.431 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:05 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:05.431 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:05 smithi082 ceph-mon[29354]: Added host smithi082 2024-06-06T07:36:05.431 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:05 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:36:06.083 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-06T07:36:06.083 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.250+0000 7f415cf36640 1 Processor -- start 2024-06-06T07:36:06.083 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.250+0000 7f415cf36640 1 -- start start 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.250+0000 7f415cf36640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158106080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.250+0000 7f415cf36640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158106650 con 0x7f4158105c60 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.251+0000 7f415659c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158106080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.251+0000 7f415659c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158106080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33168/0 (socket says 172.21.15.82:33168) 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.251+0000 7f415659c640 1 -- 172.21.15.82:0/1430212106 learned_addr learned my addr 172.21.15.82:0/1430212106 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.251+0000 7f415659c640 1 -- 172.21.15.82:0/1430212106 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158106e60 con 0x7f4158105c60 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.251+0000 7f415659c640 1 --2- 172.21.15.82:0/1430212106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158106080 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4148009820 tx=0x7f414802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7057163f0b78eb server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415559a640 1 -- 172.21.15.82:0/1430212106 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f414802ad40 con 0x7f4158105c60 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415559a640 1 -- 172.21.15.82:0/1430212106 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f41480328c0 con 0x7f4158105c60 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415559a640 1 -- 172.21.15.82:0/1430212106 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41480307f0 con 0x7f4158105c60 2024-06-06T07:36:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415cf36640 1 -- 172.21.15.82:0/1430212106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 msgr2=0x7f4158106080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415cf36640 1 --2- 172.21.15.82:0/1430212106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158106080 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4148009820 tx=0x7f414802a270 comp rx=0 tx=0).stop 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415cf36640 1 -- 172.21.15.82:0/1430212106 shutdown_connections 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415cf36640 1 --2- 172.21.15.82:0/1430212106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158106080 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415cf36640 1 -- 172.21.15.82:0/1430212106 >> 172.21.15.82:0/1430212106 conn(0x7f41581011c0 msgr2=0x7f4158103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415cf36640 1 -- 172.21.15.82:0/1430212106 shutdown_connections 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.252+0000 7f415cf36640 1 -- 172.21.15.82:0/1430212106 wait complete. 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.253+0000 7f415cf36640 1 Processor -- start 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.253+0000 7f415cf36640 1 -- start start 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.253+0000 7f415cf36640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.253+0000 7f415cf36640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158199dd0 con 0x7f4158105c60 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.253+0000 7f415659c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:06.085 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.253+0000 7f415659c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33176/0 (socket says 172.21.15.82:33176) 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.253+0000 7f415659c640 1 -- 172.21.15.82:0/3924069352 learned_addr learned my addr 172.21.15.82:0/3924069352 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.254+0000 7f415659c640 1 -- 172.21.15.82:0/3924069352 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41480094d0 con 0x7f4158105c60 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.254+0000 7f415659c640 1 --2- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158199890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4148009940 tx=0x7f4148032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.254+0000 7f41477fe640 1 -- 172.21.15.82:0/3924069352 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4148041070 con 0x7f4158105c60 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.254+0000 7f41477fe640 1 -- 172.21.15.82:0/3924069352 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4148030cc0 con 0x7f4158105c60 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.254+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4158199fd0 con 0x7f4158105c60 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.254+0000 7f41477fe640 1 -- 172.21.15.82:0/3924069352 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f414803c690 con 0x7f4158105c60 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.254+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415819a3f0 con 0x7f4158105c60 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.255+0000 7f41477fe640 1 -- 172.21.15.82:0/3924069352 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f414803c880 con 0x7f4158105c60 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.255+0000 7f41477fe640 1 --2- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f4140037860 0x7f4140039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.255+0000 7f41477fe640 1 -- 172.21.15.82:0/3924069352 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f414803d480 con 0x7f4158105c60 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.255+0000 7f4155d9b640 1 --2- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f4140037860 0x7f4140039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:06.086 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.255+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4158061ae0 con 0x7f4158105c60 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.255+0000 7f4155d9b640 1 --2- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f4140037860 0x7f4140039d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f414c0098c0 tx=0x7f414c006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.260+0000 7f41477fe640 1 -- 172.21.15.82:0/3924069352 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4148031dd0 con 0x7f4158105c60 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.447+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f41581022f0 con 0x7f4140037860 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.454+0000 7f41477fe640 1 -- 172.21.15.82:0/3924069352 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f41581022f0 con 0x7f4140037860 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f4140037860 msgr2=0x7f4140039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 --2- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f4140037860 0x7f4140039d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f414c0098c0 tx=0x7f414c006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 msgr2=0x7f4158199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 --2- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158199890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4148009940 tx=0x7f4148032c60 comp rx=0 tx=0).stop 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 shutdown_connections 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 --2- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f4140037860 0x7f4140039d20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 --2- 172.21.15.82:0/3924069352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4158105c60 0x7f4158199890 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 >> 172.21.15.82:0/3924069352 conn(0x7f41581011c0 msgr2=0x7f4158101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:06.087 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 shutdown_connections 2024-06-06T07:36:06.088 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:05.457+0000 7f415cf36640 1 -- 172.21.15.82:0/3924069352 wait complete. 2024-06-06T07:36:06.088 INFO:teuthology.orchestra.run.smithi082.stdout:Deploying mgr service with default placement... 2024-06-06T07:36:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:06 smithi082 ceph-mon[29354]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:06 smithi082 ceph-mon[29354]: Saving service mon spec with placement count:5 2024-06-06T07:36:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:06 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:08.167 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:07 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:08.167 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:07 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:08.167 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:07 smithi082 ceph-mon[29354]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:08.168 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:07 smithi082 ceph-mon[29354]: Saving service mgr spec with placement count:2 2024-06-06T07:36:08.168 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:07 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:08.714 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.271+0000 7f351113b640 1 Processor -- start 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.271+0000 7f351113b640 1 -- start start 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.272+0000 7f351113b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.272+0000 7f351113b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c105500 con 0x7f350c104b10 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.272+0000 7f350ad9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.272+0000 7f350ad9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c104f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33180/0 (socket says 172.21.15.82:33180) 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.272+0000 7f350ad9d640 1 -- 172.21.15.82:0/1927411577 learned_addr learned my addr 172.21.15.82:0/1927411577 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.272+0000 7f350ad9d640 1 -- 172.21.15.82:0/1927411577 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c105d10 con 0x7f350c104b10 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.273+0000 7f350ad9d640 1 --2- 172.21.15.82:0/1927411577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c104f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009820 tx=0x7f34fc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398742a7d835fadf server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.273+0000 7f350a59c640 1 -- 172.21.15.82:0/1927411577 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34fc02ad40 con 0x7f350c104b10 2024-06-06T07:36:08.715 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.273+0000 7f350a59c640 1 -- 172.21.15.82:0/1927411577 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34fc032900 con 0x7f350c104b10 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.273+0000 7f350a59c640 1 -- 172.21.15.82:0/1927411577 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34fc030860 con 0x7f350c104b10 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.273+0000 7f351113b640 1 -- 172.21.15.82:0/1927411577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 msgr2=0x7f350c104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.273+0000 7f351113b640 1 --2- 172.21.15.82:0/1927411577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c104f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009820 tx=0x7f34fc02a270 comp rx=0 tx=0).stop 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.273+0000 7f351113b640 1 -- 172.21.15.82:0/1927411577 shutdown_connections 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.273+0000 7f351113b640 1 --2- 172.21.15.82:0/1927411577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c104f30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.273+0000 7f351113b640 1 -- 172.21.15.82:0/1927411577 >> 172.21.15.82:0/1927411577 conn(0x7f350c1000d0 msgr2=0x7f350c1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.274+0000 7f351113b640 1 -- 172.21.15.82:0/1927411577 shutdown_connections 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.274+0000 7f351113b640 1 -- 172.21.15.82:0/1927411577 wait complete. 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.274+0000 7f351113b640 1 Processor -- start 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.274+0000 7f351113b640 1 -- start start 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.274+0000 7f351113b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c18fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.274+0000 7f351113b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c1901c0 con 0x7f350c104b10 2024-06-06T07:36:08.716 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f350ad9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c18fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f350ad9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c18fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33188/0 (socket says 172.21.15.82:33188) 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f350ad9d640 1 -- 172.21.15.82:0/4049975439 learned_addr learned my addr 172.21.15.82:0/4049975439 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f350ad9d640 1 -- 172.21.15.82:0/4049975439 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc0094d0 con 0x7f350c104b10 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f350ad9d640 1 --2- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c18fc80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009940 tx=0x7f34fc032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f3508d99640 1 -- 172.21.15.82:0/4049975439 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34fc032c80 con 0x7f350c104b10 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c18e400 con 0x7f350c104b10 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f3508d99640 1 -- 172.21.15.82:0/4049975439 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34fc032de0 con 0x7f350c104b10 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f3508d99640 1 -- 172.21.15.82:0/4049975439 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34fc03a730 con 0x7f350c104b10 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.275+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c18e8a0 con 0x7f350c104b10 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.276+0000 7f3508d99640 1 -- 172.21.15.82:0/4049975439 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f34fc03a9f0 con 0x7f350c104b10 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.276+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c061ae0 con 0x7f350c104b10 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.276+0000 7f3508d99640 1 --2- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f34f8037860 0x7f34f8039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:08.717 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.276+0000 7f3508d99640 1 -- 172.21.15.82:0/4049975439 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f34fc06aab0 con 0x7f350c104b10 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.277+0000 7f3503fff640 1 --2- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f34f8037860 0x7f34f8039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.277+0000 7f3503fff640 1 --2- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f34f8037860 0x7f34f8039d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f34f40098c0 tx=0x7f34f4006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.281+0000 7f3508d99640 1 -- 172.21.15.82:0/4049975439 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f34fc0394f0 con 0x7f350c104b10 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.521+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f350c193950 con 0x7f34f8037860 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.526+0000 7f3508d99640 1 -- 172.21.15.82:0/4049975439 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f350c193950 con 0x7f34f8037860 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f34f8037860 msgr2=0x7f34f8039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 --2- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f34f8037860 0x7f34f8039d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f34f40098c0 tx=0x7f34f4006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 msgr2=0x7f350c18fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 --2- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c18fc80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009940 tx=0x7f34fc032670 comp rx=0 tx=0).stop 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 shutdown_connections 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 --2- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f34f8037860 0x7f34f8039d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 --2- 172.21.15.82:0/4049975439 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f350c104b10 0x7f350c18fc80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 >> 172.21.15.82:0/4049975439 conn(0x7f350c1000d0 msgr2=0x7f350c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:08.718 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 shutdown_connections 2024-06-06T07:36:08.719 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:07.528+0000 7f351113b640 1 -- 172.21.15.82:0/4049975439 wait complete. 2024-06-06T07:36:08.719 INFO:teuthology.orchestra.run.smithi082.stdout:Deploying crash service with default placement... 2024-06-06T07:36:09.345 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:09 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:10.003 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.128+0000 7fee0e595640 1 Processor -- start 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.128+0000 7fee0e595640 1 -- start start 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.128+0000 7fee0e595640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08108a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.128+0000 7fee0e595640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee08079e40 con 0x7fee08106680 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.128+0000 7fee07fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.128+0000 7fee07fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33198/0 (socket says 172.21.15.82:33198) 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.128+0000 7fee07fff640 1 -- 172.21.15.82:0/3577955124 learned_addr learned my addr 172.21.15.82:0/3577955124 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.129+0000 7fee07fff640 1 -- 172.21.15.82:0/3577955124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee08108fb0 con 0x7fee08106680 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.129+0000 7fee07fff640 1 --2- 172.21.15.82:0/3577955124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08108a70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009820 tx=0x7fedf802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e9274a8e83336f server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee06ffd640 1 -- 172.21.15.82:0/3577955124 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf802ad40 con 0x7fee08106680 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee06ffd640 1 -- 172.21.15.82:0/3577955124 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedf8032900 con 0x7fee08106680 2024-06-06T07:36:10.004 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee06ffd640 1 -- 172.21.15.82:0/3577955124 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf8030860 con 0x7fee08106680 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee0e595640 1 -- 172.21.15.82:0/3577955124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 msgr2=0x7fee08108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee0e595640 1 --2- 172.21.15.82:0/3577955124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08108a70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009820 tx=0x7fedf802a270 comp rx=0 tx=0).stop 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee0e595640 1 -- 172.21.15.82:0/3577955124 shutdown_connections 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee0e595640 1 --2- 172.21.15.82:0/3577955124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08108a70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee0e595640 1 -- 172.21.15.82:0/3577955124 >> 172.21.15.82:0/3577955124 conn(0x7fee081000d0 msgr2=0x7fee081024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee0e595640 1 -- 172.21.15.82:0/3577955124 shutdown_connections 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.130+0000 7fee0e595640 1 -- 172.21.15.82:0/3577955124 wait complete. 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.131+0000 7fee0e595640 1 Processor -- start 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.131+0000 7fee0e595640 1 -- start start 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.131+0000 7fee0e595640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.131+0000 7fee0e595640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081063f0 con 0x7fee08106680 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee07fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee07fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33210/0 (socket says 172.21.15.82:33210) 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee07fff640 1 -- 172.21.15.82:0/3631213499 learned_addr learned my addr 172.21.15.82:0/3631213499 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee07fff640 1 -- 172.21.15.82:0/3631213499 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80094d0 con 0x7fee08106680 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee07fff640 1 --2- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08105eb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fedf80097f0 tx=0x7fedf8030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee057fa640 1 -- 172.21.15.82:0/3631213499 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf8040070 con 0x7fee08106680 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee08104630 con 0x7fee08106680 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee057fa640 1 -- 172.21.15.82:0/3631213499 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedf8032dd0 con 0x7fee08106680 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee057fa640 1 -- 172.21.15.82:0/3631213499 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf803a800 con 0x7fee08106680 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.132+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee08104ad0 con 0x7fee08106680 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.133+0000 7fee057fa640 1 -- 172.21.15.82:0/3631213499 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fedf803a960 con 0x7fee08106680 2024-06-06T07:36:10.006 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.133+0000 7fee057fa640 1 --2- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fedf0037860 0x7fedf0039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.133+0000 7fee057fa640 1 -- 172.21.15.82:0/3631213499 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fedf806b620 con 0x7fee08106680 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.133+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee08061ae0 con 0x7fee08106680 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.134+0000 7fee077fe640 1 --2- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fedf0037860 0x7fedf0039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.134+0000 7fee077fe640 1 --2- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fedf0037860 0x7fedf0039d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0098c0 tx=0x7fedfc006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.138+0000 7fee057fa640 1 -- 172.21.15.82:0/3631213499 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fedf8039b70 con 0x7fee08106680 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.344+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fee08105020 con 0x7fedf0037860 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.352+0000 7fee057fa640 1 -- 172.21.15.82:0/3631213499 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fee08105020 con 0x7fedf0037860 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fedf0037860 msgr2=0x7fedf0039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 --2- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fedf0037860 0x7fedf0039d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0098c0 tx=0x7fedfc006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 msgr2=0x7fee08105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 --2- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08105eb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fedf80097f0 tx=0x7fedf8030e90 comp rx=0 tx=0).stop 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 shutdown_connections 2024-06-06T07:36:10.007 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 --2- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fedf0037860 0x7fedf0039d20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:10.008 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 --2- 172.21.15.82:0/3631213499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee08106680 0x7fee08105eb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:10.008 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 >> 172.21.15.82:0/3631213499 conn(0x7fee081000d0 msgr2=0x7fee08100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:10.008 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 shutdown_connections 2024-06-06T07:36:10.008 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:09.354+0000 7fee0e595640 1 -- 172.21.15.82:0/3631213499 wait complete. 2024-06-06T07:36:10.008 INFO:teuthology.orchestra.run.smithi082.stdout:Deploying ceph-exporter service with default placement... 2024-06-06T07:36:10.418 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:10 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:10.418 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:10 smithi082 ceph-mon[29354]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:10.419 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:10 smithi082 ceph-mon[29354]: Saving service crash spec with placement * 2024-06-06T07:36:10.419 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:10 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:11.664 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:11 smithi082 ceph-mon[29354]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:11.664 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:11 smithi082 ceph-mon[29354]: Saving service ceph-exporter spec with placement * 2024-06-06T07:36:11.664 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:11 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:12.074 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-06-06T07:36:12.074 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.418+0000 7f136b088640 1 Processor -- start 2024-06-06T07:36:12.074 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.418+0000 7f136b088640 1 -- start start 2024-06-06T07:36:12.074 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.418+0000 7f136b088640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13641066f0 0x7f1364108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:12.074 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.418+0000 7f136b088640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1364079e40 con 0x7f13641066f0 2024-06-06T07:36:12.074 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.418+0000 7f1368e24640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13641066f0 0x7f1364108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.418+0000 7f1368e24640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13641066f0 0x7f1364108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33226/0 (socket says 172.21.15.82:33226) 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.418+0000 7f1368e24640 1 -- 172.21.15.82:0/586479604 learned_addr learned my addr 172.21.15.82:0/586479604 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.418+0000 7f1368e24640 1 -- 172.21.15.82:0/586479604 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1364109020 con 0x7f13641066f0 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.419+0000 7f1368e24640 1 --2- 172.21.15.82:0/586479604 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13641066f0 0x7f1364108ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1358009820 tx=0x7f135802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce01f7b98e58ebec server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.419+0000 7f13637fe640 1 -- 172.21.15.82:0/586479604 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f135802ad40 con 0x7f13641066f0 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.419+0000 7f13637fe640 1 -- 172.21.15.82:0/586479604 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1358032900 con 0x7f13641066f0 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.419+0000 7f13637fe640 1 -- 172.21.15.82:0/586479604 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1358030860 con 0x7f13641066f0 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.420+0000 7f136b088640 1 -- 172.21.15.82:0/586479604 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13641066f0 msgr2=0x7f1364108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.420+0000 7f136b088640 1 --2- 172.21.15.82:0/586479604 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13641066f0 0x7f1364108ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1358009820 tx=0x7f135802a270 comp rx=0 tx=0).stop 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.420+0000 7f136b088640 1 -- 172.21.15.82:0/586479604 shutdown_connections 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.420+0000 7f136b088640 1 --2- 172.21.15.82:0/586479604 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13641066f0 0x7f1364108ae0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.420+0000 7f136b088640 1 -- 172.21.15.82:0/586479604 >> 172.21.15.82:0/586479604 conn(0x7f13641000e0 msgr2=0x7f1364102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.420+0000 7f136b088640 1 -- 172.21.15.82:0/586479604 shutdown_connections 2024-06-06T07:36:12.075 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.420+0000 7f136b088640 1 -- 172.21.15.82:0/586479604 wait complete. 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.420+0000 7f136b088640 1 Processor -- start 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.420+0000 7f136b088640 1 -- start start 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f136b088640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1364105ee0 0x7f13641046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f136b088640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1364106300 con 0x7f1364105ee0 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f1368e24640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1364105ee0 0x7f13641046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f1368e24640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1364105ee0 0x7f13641046a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33232/0 (socket says 172.21.15.82:33232) 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f1368e24640 1 -- 172.21.15.82:0/3500451729 learned_addr learned my addr 172.21.15.82:0/3500451729 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f1368e24640 1 -- 172.21.15.82:0/3500451729 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13580094d0 con 0x7f1364105ee0 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f1368e24640 1 --2- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1364105ee0 0x7f13641046a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f13580097f0 tx=0x7f1358032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f1361ffb640 1 -- 172.21.15.82:0/3500451729 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1358032dd0 con 0x7f1364105ee0 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f1361ffb640 1 -- 172.21.15.82:0/3500451729 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f135803b8b0 con 0x7f1364105ee0 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1364106500 con 0x7f1364105ee0 2024-06-06T07:36:12.076 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.421+0000 7f1361ffb640 1 -- 172.21.15.82:0/3500451729 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f135803a6e0 con 0x7f1364105ee0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.422+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1364104e30 con 0x7f1364105ee0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.422+0000 7f1361ffb640 1 -- 172.21.15.82:0/3500451729 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f1358039040 con 0x7f1364105ee0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.422+0000 7f1361ffb640 1 --2- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f13540378b0 0x7f1354039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.422+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1364109e80 con 0x7f1364105ee0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.423+0000 7f1361ffb640 1 -- 172.21.15.82:0/3500451729 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f135806a770 con 0x7f1364105ee0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.423+0000 7f1363fff640 1 --2- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f13540378b0 0x7f1354039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.423+0000 7f1363fff640 1 --2- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f13540378b0 0x7f1354039d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f13500098c0 tx=0x7f1350006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.427+0000 7f1361ffb640 1 -- 172.21.15.82:0/3500451729 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f135804c350 con 0x7f1364105ee0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.615+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f1364101260 con 0x7f13540378b0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.622+0000 7f1361ffb640 1 -- 172.21.15.82:0/3500451729 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f1364101260 con 0x7f13540378b0 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f13540378b0 msgr2=0x7f1354039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 --2- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f13540378b0 0x7f1354039d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f13500098c0 tx=0x7f1350006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:12.077 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1364105ee0 msgr2=0x7f13641046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:12.078 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 --2- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1364105ee0 0x7f13641046a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f13580097f0 tx=0x7f1358032670 comp rx=0 tx=0).stop 2024-06-06T07:36:12.078 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 shutdown_connections 2024-06-06T07:36:12.078 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 --2- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f13540378b0 0x7f1354039d70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:12.078 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 --2- 172.21.15.82:0/3500451729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1364105ee0 0x7f13641046a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:12.078 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 >> 172.21.15.82:0/3500451729 conn(0x7f13641000e0 msgr2=0x7f1364100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:12.078 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 shutdown_connections 2024-06-06T07:36:12.078 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:10.624+0000 7f136b088640 1 -- 172.21.15.82:0/3500451729 wait complete. 2024-06-06T07:36:12.078 INFO:teuthology.orchestra.run.smithi082.stdout:Deploying prometheus service with default placement... 2024-06-06T07:36:13.246 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-06-06T07:36:13.246 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.477+0000 7f90d352f640 1 Processor -- start 2024-06-06T07:36:13.246 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.477+0000 7f90d352f640 1 -- start start 2024-06-06T07:36:13.246 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.477+0000 7f90d352f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:13.246 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.477+0000 7f90d352f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc106630 con 0x7f90cc105c40 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.478+0000 7f90d12cb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.478+0000 7f90d12cb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33242/0 (socket says 172.21.15.82:33242) 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.478+0000 7f90d12cb640 1 -- 172.21.15.82:0/892966848 learned_addr learned my addr 172.21.15.82:0/892966848 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.478+0000 7f90d12cb640 1 -- 172.21.15.82:0/892966848 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc106e40 con 0x7f90cc105c40 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.479+0000 7f90d12cb640 1 --2- 172.21.15.82:0/892966848 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc106060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009820 tx=0x7f90c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2ae0700b3ef05d server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.479+0000 7f90c3fff640 1 -- 172.21.15.82:0/892966848 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c802ad40 con 0x7f90cc105c40 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.479+0000 7f90c3fff640 1 -- 172.21.15.82:0/892966848 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90c8032900 con 0x7f90cc105c40 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.479+0000 7f90c3fff640 1 -- 172.21.15.82:0/892966848 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c8030860 con 0x7f90cc105c40 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.479+0000 7f90d352f640 1 -- 172.21.15.82:0/892966848 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 msgr2=0x7f90cc106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.479+0000 7f90d352f640 1 --2- 172.21.15.82:0/892966848 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc106060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009820 tx=0x7f90c802a270 comp rx=0 tx=0).stop 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.479+0000 7f90d352f640 1 -- 172.21.15.82:0/892966848 shutdown_connections 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.479+0000 7f90d352f640 1 --2- 172.21.15.82:0/892966848 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc106060 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.479+0000 7f90d352f640 1 -- 172.21.15.82:0/892966848 >> 172.21.15.82:0/892966848 conn(0x7f90cc1011c0 msgr2=0x7f90cc103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:13.247 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.480+0000 7f90d352f640 1 -- 172.21.15.82:0/892966848 shutdown_connections 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.480+0000 7f90d352f640 1 -- 172.21.15.82:0/892966848 wait complete. 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.480+0000 7f90d352f640 1 Processor -- start 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.480+0000 7f90d352f640 1 -- start start 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.480+0000 7f90d352f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc07bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.480+0000 7f90d352f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc07bff0 con 0x7f90cc105c40 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.480+0000 7f90d12cb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc07bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.480+0000 7f90d12cb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc07bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33246/0 (socket says 172.21.15.82:33246) 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.480+0000 7f90d12cb640 1 -- 172.21.15.82:0/818862621 learned_addr learned my addr 172.21.15.82:0/818862621 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.481+0000 7f90d12cb640 1 -- 172.21.15.82:0/818862621 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c80094d0 con 0x7f90cc105c40 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.481+0000 7f90d12cb640 1 --2- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc07bab0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009940 tx=0x7f90c8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.481+0000 7f90c27fc640 1 -- 172.21.15.82:0/818862621 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c8032c80 con 0x7f90cc105c40 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.481+0000 7f90c27fc640 1 -- 172.21.15.82:0/818862621 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90c8032de0 con 0x7f90cc105c40 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.481+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc07a230 con 0x7f90cc105c40 2024-06-06T07:36:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.481+0000 7f90c27fc640 1 -- 172.21.15.82:0/818862621 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c803a730 con 0x7f90cc105c40 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.481+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc07a6d0 con 0x7f90cc105c40 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.482+0000 7f90c27fc640 1 -- 172.21.15.82:0/818862621 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f90c803a9f0 con 0x7f90cc105c40 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.482+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc10a070 con 0x7f90cc105c40 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.482+0000 7f90c27fc640 1 --2- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f90b8037860 0x7f90b8039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.482+0000 7f90c27fc640 1 -- 172.21.15.82:0/818862621 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f90c806ab00 con 0x7f90cc105c40 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.482+0000 7f90d0aca640 1 --2- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f90b8037860 0x7f90b8039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.483+0000 7f90d0aca640 1 --2- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f90b8037860 0x7f90b8039d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0098c0 tx=0x7f90bc006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.487+0000 7f90c27fc640 1 -- 172.21.15.82:0/818862621 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f90c8039b90 con 0x7f90cc105c40 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.668+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f90cc07aec0 con 0x7f90b8037860 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.673+0000 7f90c27fc640 1 -- 172.21.15.82:0/818862621 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f90cc07aec0 con 0x7f90b8037860 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f90b8037860 msgr2=0x7f90b8039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 --2- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f90b8037860 0x7f90b8039d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0098c0 tx=0x7f90bc006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 msgr2=0x7f90cc07bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 --2- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc07bab0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009940 tx=0x7f90c8032670 comp rx=0 tx=0).stop 2024-06-06T07:36:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 shutdown_connections 2024-06-06T07:36:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 --2- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f90b8037860 0x7f90b8039d20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 --2- 172.21.15.82:0/818862621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90cc105c40 0x7f90cc07bab0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 >> 172.21.15.82:0/818862621 conn(0x7f90cc1011c0 msgr2=0x7f90cc101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 shutdown_connections 2024-06-06T07:36:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:12.675+0000 7f90d352f640 1 -- 172.21.15.82:0/818862621 wait complete. 2024-06-06T07:36:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:Deploying grafana service with default placement... 2024-06-06T07:36:13.813 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:13 smithi082 ceph-mon[29354]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:13.813 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:13 smithi082 ceph-mon[29354]: Saving service prometheus spec with placement count:1 2024-06-06T07:36:13.813 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:13 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:14.451 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.617+0000 7fd147c9b640 1 Processor -- start 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.618+0000 7fd147c9b640 1 -- start start 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.618+0000 7fd147c9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.618+0000 7fd147c9b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14007a8a0 con 0x7fd14007be70 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.618+0000 7fd145a37640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.618+0000 7fd145a37640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33260/0 (socket says 172.21.15.82:33260) 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.618+0000 7fd145a37640 1 -- 172.21.15.82:0/385869166 learned_addr learned my addr 172.21.15.82:0/385869166 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.618+0000 7fd145a37640 1 -- 172.21.15.82:0/385869166 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14007a9e0 con 0x7fd14007be70 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.619+0000 7fd145a37640 1 --2- 172.21.15.82:0/385869166 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14007a2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009820 tx=0x7fd13c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acbdb05d00175043 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.619+0000 7fd144a35640 1 -- 172.21.15.82:0/385869166 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd13c02ad40 con 0x7fd14007be70 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.619+0000 7fd144a35640 1 -- 172.21.15.82:0/385869166 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd13c032900 con 0x7fd14007be70 2024-06-06T07:36:14.452 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.619+0000 7fd144a35640 1 -- 172.21.15.82:0/385869166 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd13c030860 con 0x7fd14007be70 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.620+0000 7fd147c9b640 1 -- 172.21.15.82:0/385869166 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 msgr2=0x7fd14007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.620+0000 7fd147c9b640 1 --2- 172.21.15.82:0/385869166 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14007a2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009820 tx=0x7fd13c02a270 comp rx=0 tx=0).stop 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.620+0000 7fd147c9b640 1 -- 172.21.15.82:0/385869166 shutdown_connections 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.620+0000 7fd147c9b640 1 --2- 172.21.15.82:0/385869166 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14007a2d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.620+0000 7fd147c9b640 1 -- 172.21.15.82:0/385869166 >> 172.21.15.82:0/385869166 conn(0x7fd1401011c0 msgr2=0x7fd140103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.620+0000 7fd147c9b640 1 -- 172.21.15.82:0/385869166 shutdown_connections 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.620+0000 7fd147c9b640 1 -- 172.21.15.82:0/385869166 wait complete. 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.621+0000 7fd147c9b640 1 Processor -- start 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.621+0000 7fd147c9b640 1 -- start start 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.621+0000 7fd147c9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14019bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.621+0000 7fd147c9b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14019c040 con 0x7fd14007be70 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.621+0000 7fd145a37640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14019bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:14.453 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.621+0000 7fd145a37640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14019bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33268/0 (socket says 172.21.15.82:33268) 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.621+0000 7fd145a37640 1 -- 172.21.15.82:0/1503476640 learned_addr learned my addr 172.21.15.82:0/1503476640 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.621+0000 7fd145a37640 1 -- 172.21.15.82:0/1503476640 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c0094d0 con 0x7fd14007be70 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.622+0000 7fd145a37640 1 --2- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14019bb00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0097f0 tx=0x7fd13c030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.622+0000 7fd132ffd640 1 -- 172.21.15.82:0/1503476640 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd13c040070 con 0x7fd14007be70 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.622+0000 7fd132ffd640 1 -- 172.21.15.82:0/1503476640 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd13c032dd0 con 0x7fd14007be70 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.622+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14019c240 con 0x7fd14007be70 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.622+0000 7fd132ffd640 1 -- 172.21.15.82:0/1503476640 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd13c03a800 con 0x7fd14007be70 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.622+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14019c6e0 con 0x7fd14007be70 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.623+0000 7fd132ffd640 1 -- 172.21.15.82:0/1503476640 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fd13c03a960 con 0x7fd14007be70 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.623+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14010c040 con 0x7fd14007be70 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.623+0000 7fd132ffd640 1 --2- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd12c0378b0 0x7fd12c039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.623+0000 7fd132ffd640 1 -- 172.21.15.82:0/1503476640 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd13c06a5f0 con 0x7fd14007be70 2024-06-06T07:36:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.623+0000 7fd145236640 1 --2- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd12c0378b0 0x7fd12c039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.623+0000 7fd145236640 1 --2- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd12c0378b0 0x7fd12c039d70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd1340098c0 tx=0x7fd134006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.627+0000 7fd132ffd640 1 -- 172.21.15.82:0/1503476640 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd13c039780 con 0x7fd14007be70 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.812+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1401023a0 con 0x7fd12c0378b0 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.819+0000 7fd132ffd640 1 -- 172.21.15.82:0/1503476640 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fd1401023a0 con 0x7fd12c0378b0 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.821+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd12c0378b0 msgr2=0x7fd12c039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.821+0000 7fd147c9b640 1 --2- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd12c0378b0 0x7fd12c039d70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd1340098c0 tx=0x7fd134006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.821+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 msgr2=0x7fd14019bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.821+0000 7fd147c9b640 1 --2- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14019bb00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0097f0 tx=0x7fd13c030e90 comp rx=0 tx=0).stop 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.821+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 shutdown_connections 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.821+0000 7fd147c9b640 1 --2- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd12c0378b0 0x7fd12c039d70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.821+0000 7fd147c9b640 1 --2- 172.21.15.82:0/1503476640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd14007be70 0x7fd14019bb00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.821+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 >> 172.21.15.82:0/1503476640 conn(0x7fd1401011c0 msgr2=0x7fd140101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.821+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 shutdown_connections 2024-06-06T07:36:14.455 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:13.822+0000 7fd147c9b640 1 -- 172.21.15.82:0/1503476640 wait complete. 2024-06-06T07:36:14.456 INFO:teuthology.orchestra.run.smithi082.stdout:Deploying node-exporter service with default placement... 2024-06-06T07:36:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:14 smithi082 ceph-mon[29354]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:14 smithi082 ceph-mon[29354]: Saving service grafana spec with placement count:1 2024-06-06T07:36:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:14 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:14 smithi082 ceph-mon[29354]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.833+0000 7fb1e0d53640 1 Processor -- start 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.833+0000 7fb1e0d53640 1 -- start start 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.833+0000 7fb1e0d53640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.833+0000 7fb1e0d53640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc106630 con 0x7fb1dc105c40 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.833+0000 7fb1da59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.833+0000 7fb1da59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33284/0 (socket says 172.21.15.82:33284) 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.833+0000 7fb1da59c640 1 -- 172.21.15.82:0/2779953652 learned_addr learned my addr 172.21.15.82:0/2779953652 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.834+0000 7fb1da59c640 1 -- 172.21.15.82:0/2779953652 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc106e40 con 0x7fb1dc105c40 2024-06-06T07:36:15.583 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.834+0000 7fb1da59c640 1 --2- 172.21.15.82:0/2779953652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc106060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009820 tx=0x7fb1cc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8068a4aa2a84e666 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.834+0000 7fb1d959a640 1 -- 172.21.15.82:0/2779953652 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc02ad40 con 0x7fb1dc105c40 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.834+0000 7fb1d959a640 1 -- 172.21.15.82:0/2779953652 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1cc032900 con 0x7fb1dc105c40 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.834+0000 7fb1d959a640 1 -- 172.21.15.82:0/2779953652 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc030860 con 0x7fb1dc105c40 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.835+0000 7fb1e0d53640 1 -- 172.21.15.82:0/2779953652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 msgr2=0x7fb1dc106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.835+0000 7fb1e0d53640 1 --2- 172.21.15.82:0/2779953652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc106060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009820 tx=0x7fb1cc02a270 comp rx=0 tx=0).stop 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.835+0000 7fb1e0d53640 1 -- 172.21.15.82:0/2779953652 shutdown_connections 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.835+0000 7fb1e0d53640 1 --2- 172.21.15.82:0/2779953652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc106060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.835+0000 7fb1e0d53640 1 -- 172.21.15.82:0/2779953652 >> 172.21.15.82:0/2779953652 conn(0x7fb1dc1011c0 msgr2=0x7fb1dc103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.835+0000 7fb1e0d53640 1 -- 172.21.15.82:0/2779953652 shutdown_connections 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.835+0000 7fb1e0d53640 1 -- 172.21.15.82:0/2779953652 wait complete. 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.836+0000 7fb1e0d53640 1 Processor -- start 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.836+0000 7fb1e0d53640 1 -- start start 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.836+0000 7fb1e0d53640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:15.584 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.836+0000 7fb1e0d53640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc199de0 con 0x7fb1dc105c40 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.836+0000 7fb1da59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.836+0000 7fb1da59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc1998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33294/0 (socket says 172.21.15.82:33294) 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.836+0000 7fb1da59c640 1 -- 172.21.15.82:0/1374181900 learned_addr learned my addr 172.21.15.82:0/1374181900 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.836+0000 7fb1da59c640 1 -- 172.21.15.82:0/1374181900 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc0094d0 con 0x7fb1dc105c40 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.837+0000 7fb1da59c640 1 --2- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc1998a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009940 tx=0x7fb1cc032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.837+0000 7fb1cb7fe640 1 -- 172.21.15.82:0/1374181900 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc032c80 con 0x7fb1dc105c40 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.837+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc199fe0 con 0x7fb1dc105c40 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.837+0000 7fb1cb7fe640 1 -- 172.21.15.82:0/1374181900 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1cc032de0 con 0x7fb1dc105c40 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.837+0000 7fb1cb7fe640 1 -- 172.21.15.82:0/1374181900 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc03a730 con 0x7fb1dc105c40 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.837+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc19a480 con 0x7fb1dc105c40 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.838+0000 7fb1cb7fe640 1 -- 172.21.15.82:0/1374181900 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fb1cc03a9f0 con 0x7fb1dc105c40 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.838+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc10a070 con 0x7fb1dc105c40 2024-06-06T07:36:15.585 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.838+0000 7fb1cb7fe640 1 --2- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fb1c4037860 0x7fb1c4039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.838+0000 7fb1cb7fe640 1 -- 172.21.15.82:0/1374181900 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb1cc06a8f0 con 0x7fb1dc105c40 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.838+0000 7fb1d9d9b640 1 --2- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fb1c4037860 0x7fb1c4039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.838+0000 7fb1d9d9b640 1 --2- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fb1c4037860 0x7fb1c4039d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00098c0 tx=0x7fb1d0006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:14.842+0000 7fb1cb7fe640 1 -- 172.21.15.82:0/1374181900 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb1cc039b90 con 0x7fb1dc105c40 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.023+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1dc1022d0 con 0x7fb1c4037860 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.028+0000 7fb1cb7fe640 1 -- 172.21.15.82:0/1374181900 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fb1dc1022d0 con 0x7fb1c4037860 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fb1c4037860 msgr2=0x7fb1c4039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 --2- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fb1c4037860 0x7fb1c4039d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00098c0 tx=0x7fb1d0006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 msgr2=0x7fb1dc1998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 --2- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc1998a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009940 tx=0x7fb1cc032670 comp rx=0 tx=0).stop 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 shutdown_connections 2024-06-06T07:36:15.586 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 --2- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fb1c4037860 0x7fb1c4039d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:15.587 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 --2- 172.21.15.82:0/1374181900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1dc105c40 0x7fb1dc1998a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:15.587 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 >> 172.21.15.82:0/1374181900 conn(0x7fb1dc1011c0 msgr2=0x7fb1dc101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:15.587 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 shutdown_connections 2024-06-06T07:36:15.587 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:15.030+0000 7fb1e0d53640 1 -- 172.21.15.82:0/1374181900 wait complete. 2024-06-06T07:36:15.587 INFO:teuthology.orchestra.run.smithi082.stdout:Deploying alertmanager service with default placement... 2024-06-06T07:36:16.206 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:16 smithi082 ceph-mon[29354]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:16.206 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:16 smithi082 ceph-mon[29354]: Saving service node-exporter spec with placement * 2024-06-06T07:36:16.207 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:16 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:16.847 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.008+0000 7fd752b73640 1 Processor -- start 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.008+0000 7fd752b73640 1 -- start start 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.008+0000 7fd752b73640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c108ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.008+0000 7fd752b73640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c079e00 con 0x7fd74c1066c0 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.008+0000 7fd75090f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c108ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.008+0000 7fd75090f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c108ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52630/0 (socket says 172.21.15.82:52630) 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.008+0000 7fd75090f640 1 -- 172.21.15.82:0/3196617804 learned_addr learned my addr 172.21.15.82:0/3196617804 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.008+0000 7fd75090f640 1 -- 172.21.15.82:0/3196617804 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c108ff0 con 0x7fd74c1066c0 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.009+0000 7fd75090f640 1 --2- 172.21.15.82:0/3196617804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c108ab0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd740009820 tx=0x7fd74002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9965afe0e06762ee server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.009+0000 7fd74b7fe640 1 -- 172.21.15.82:0/3196617804 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd74002ad40 con 0x7fd74c1066c0 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.009+0000 7fd74b7fe640 1 -- 172.21.15.82:0/3196617804 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd740032900 con 0x7fd74c1066c0 2024-06-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.009+0000 7fd74b7fe640 1 -- 172.21.15.82:0/3196617804 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd740030860 con 0x7fd74c1066c0 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.010+0000 7fd752b73640 1 -- 172.21.15.82:0/3196617804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 msgr2=0x7fd74c108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.010+0000 7fd752b73640 1 --2- 172.21.15.82:0/3196617804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c108ab0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd740009820 tx=0x7fd74002a270 comp rx=0 tx=0).stop 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.010+0000 7fd752b73640 1 -- 172.21.15.82:0/3196617804 shutdown_connections 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.010+0000 7fd752b73640 1 --2- 172.21.15.82:0/3196617804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c108ab0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.010+0000 7fd752b73640 1 -- 172.21.15.82:0/3196617804 >> 172.21.15.82:0/3196617804 conn(0x7fd74c1000d0 msgr2=0x7fd74c1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.010+0000 7fd752b73640 1 -- 172.21.15.82:0/3196617804 shutdown_connections 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.010+0000 7fd752b73640 1 -- 172.21.15.82:0/3196617804 wait complete. 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.011+0000 7fd752b73640 1 Processor -- start 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.011+0000 7fd752b73640 1 -- start start 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.011+0000 7fd752b73640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.011+0000 7fd752b73640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c199dd0 con 0x7fd74c1066c0 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.011+0000 7fd75090f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.011+0000 7fd75090f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52636/0 (socket says 172.21.15.82:52636) 2024-06-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.011+0000 7fd75090f640 1 -- 172.21.15.82:0/316378175 learned_addr learned my addr 172.21.15.82:0/316378175 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.011+0000 7fd75090f640 1 -- 172.21.15.82:0/316378175 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7400094d0 con 0x7fd74c1066c0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.011+0000 7fd75090f640 1 --2- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c199890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd7400097f0 tx=0x7fd740030e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.012+0000 7fd749ffb640 1 -- 172.21.15.82:0/316378175 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd740032c80 con 0x7fd74c1066c0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.012+0000 7fd749ffb640 1 -- 172.21.15.82:0/316378175 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd740032de0 con 0x7fd74c1066c0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.012+0000 7fd749ffb640 1 -- 172.21.15.82:0/316378175 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd74003a7c0 con 0x7fd74c1066c0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.012+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c199fd0 con 0x7fd74c1066c0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.012+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c19a470 con 0x7fd74c1066c0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.012+0000 7fd749ffb640 1 -- 172.21.15.82:0/316378175 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fd74003a9c0 con 0x7fd74c1066c0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.013+0000 7fd749ffb640 1 --2- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd73c037860 0x7fd73c039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.013+0000 7fd749ffb640 1 -- 172.21.15.82:0/316378175 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd74006a970 con 0x7fd74c1066c0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.013+0000 7fd74bfff640 1 --2- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd73c037860 0x7fd73c039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.013+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74c061ae0 con 0x7fd74c1066c0 2024-06-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.013+0000 7fd74bfff640 1 --2- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd73c037860 0x7fd73c039d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd7380098c0 tx=0x7fd738006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.018+0000 7fd749ffb640 1 -- 172.21.15.82:0/316378175 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd740043310 con 0x7fd74c1066c0 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.206+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fd74c101210 con 0x7fd73c037860 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.211+0000 7fd749ffb640 1 -- 172.21.15.82:0/316378175 <== mgr.14116 v2:172.21.15.82:6800/1662212559 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fd74c101210 con 0x7fd73c037860 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd73c037860 msgr2=0x7fd73c039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 --2- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd73c037860 0x7fd73c039d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd7380098c0 tx=0x7fd738006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 msgr2=0x7fd74c199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 --2- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c199890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd7400097f0 tx=0x7fd740030e20 comp rx=0 tx=0).stop 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 shutdown_connections 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 --2- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fd73c037860 0x7fd73c039d20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 --2- 172.21.15.82:0/316378175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd74c1066c0 0x7fd74c199890 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 >> 172.21.15.82:0/316378175 conn(0x7fd74c1000d0 msgr2=0x7fd74c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 shutdown_connections 2024-06-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:16.213+0000 7fd752b73640 1 -- 172.21.15.82:0/316378175 wait complete. 2024-06-06T07:36:17.417 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:17 smithi082 ceph-mon[29354]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:17.417 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:17 smithi082 ceph-mon[29354]: Saving service alertmanager spec with placement count:1 2024-06-06T07:36:17.417 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:17 smithi082 ceph-mon[29354]: from='mgr.14116 172.21.15.82:0/2100426720' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:17.417 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:17 smithi082 ceph-mon[29354]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:18.231 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.235+0000 7f69148be640 1 Processor -- start 2024-06-06T07:36:18.236 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.235+0000 7f69148be640 1 -- start start 2024-06-06T07:36:18.236 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.235+0000 7f69148be640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f6910106080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:18.236 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.235+0000 7f69148be640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6910106650 con 0x7f6910105c60 2024-06-06T07:36:18.236 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.235+0000 7f690e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f6910106080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:18.236 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.235+0000 7f690e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f6910106080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52640/0 (socket says 172.21.15.82:52640) 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.235+0000 7f690e59c640 1 -- 172.21.15.82:0/3127570053 learned_addr learned my addr 172.21.15.82:0/3127570053 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.236+0000 7f690e59c640 1 -- 172.21.15.82:0/3127570053 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6910106e60 con 0x7f6910105c60 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.236+0000 7f690e59c640 1 --2- 172.21.15.82:0/3127570053 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f6910106080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6900009820 tx=0x7f690002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db570a0cf151003c server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.236+0000 7f690d59a640 1 -- 172.21.15.82:0/3127570053 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f690002ad40 con 0x7f6910105c60 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.236+0000 7f690d59a640 1 -- 172.21.15.82:0/3127570053 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6900032900 con 0x7f6910105c60 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.236+0000 7f690d59a640 1 -- 172.21.15.82:0/3127570053 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6900030860 con 0x7f6910105c60 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.237+0000 7f69148be640 1 -- 172.21.15.82:0/3127570053 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 msgr2=0x7f6910106080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.237+0000 7f69148be640 1 --2- 172.21.15.82:0/3127570053 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f6910106080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6900009820 tx=0x7f690002a270 comp rx=0 tx=0).stop 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.237+0000 7f69148be640 1 -- 172.21.15.82:0/3127570053 shutdown_connections 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.237+0000 7f69148be640 1 --2- 172.21.15.82:0/3127570053 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f6910106080 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.237+0000 7f69148be640 1 -- 172.21.15.82:0/3127570053 >> 172.21.15.82:0/3127570053 conn(0x7f69101011c0 msgr2=0x7f6910103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.237+0000 7f69148be640 1 -- 172.21.15.82:0/3127570053 shutdown_connections 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.237+0000 7f69148be640 1 -- 172.21.15.82:0/3127570053 wait complete. 2024-06-06T07:36:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.238+0000 7f69148be640 1 Processor -- start 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.238+0000 7f69148be640 1 -- start start 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.238+0000 7f69148be640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f69101998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.238+0000 7f69148be640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6910199e00 con 0x7f6910105c60 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.238+0000 7f690e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f69101998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.238+0000 7f690e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f69101998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52646/0 (socket says 172.21.15.82:52646) 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.238+0000 7f690e59c640 1 -- 172.21.15.82:0/2781774936 learned_addr learned my addr 172.21.15.82:0/2781774936 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.238+0000 7f690e59c640 1 -- 172.21.15.82:0/2781774936 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69000094d0 con 0x7f6910105c60 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.239+0000 7f690e59c640 1 --2- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f69101998c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6900009940 tx=0x7f6900032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.239+0000 7f68ff7fe640 1 -- 172.21.15.82:0/2781774936 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6900032c80 con 0x7f6910105c60 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.239+0000 7f68ff7fe640 1 -- 172.21.15.82:0/2781774936 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6900032de0 con 0x7f6910105c60 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.239+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691019a000 con 0x7f6910105c60 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.239+0000 7f68ff7fe640 1 -- 172.21.15.82:0/2781774936 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f690003a730 con 0x7f6910105c60 2024-06-06T07:36:18.238 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.239+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691019a4a0 con 0x7f6910105c60 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.240+0000 7f68ff7fe640 1 -- 172.21.15.82:0/2781774936 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f690003a9a0 con 0x7f6910105c60 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.240+0000 7f68ff7fe640 1 --2- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f68f80378b0 0x7f68f8039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.240+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691010a090 con 0x7f6910105c60 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.240+0000 7f68ff7fe640 1 -- 172.21.15.82:0/2781774936 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f690006aba0 con 0x7f6910105c60 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.240+0000 7f690dd9b640 1 --2- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f68f80378b0 0x7f68f8039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.240+0000 7f690dd9b640 1 --2- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f68f80378b0 0x7f68f8039d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f69040098c0 tx=0x7f6904006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.244+0000 7f68ff7fe640 1 -- 172.21.15.82:0/2781774936 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6900039d50 con 0x7f6910105c60 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.416+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f691010a2d0 con 0x7f6910105c60 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.420+0000 7f68ff7fe640 1 -- 172.21.15.82:0/2781774936 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f6900039310 con 0x7f6910105c60 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.422+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f68f80378b0 msgr2=0x7f68f8039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.422+0000 7f69148be640 1 --2- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f68f80378b0 0x7f68f8039d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f69040098c0 tx=0x7f6904006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:18.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.422+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 msgr2=0x7f69101998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:18.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.422+0000 7f69148be640 1 --2- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f69101998c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6900009940 tx=0x7f6900032670 comp rx=0 tx=0).stop 2024-06-06T07:36:18.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.422+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 shutdown_connections 2024-06-06T07:36:18.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.422+0000 7f69148be640 1 --2- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f68f80378b0 0x7f68f8039d70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:18.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.422+0000 7f69148be640 1 --2- 172.21.15.82:0/2781774936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6910105c60 0x7f69101998c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:18.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.422+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 >> 172.21.15.82:0/2781774936 conn(0x7f69101011c0 msgr2=0x7f6910101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:18.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.423+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 shutdown_connections 2024-06-06T07:36:18.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:17.423+0000 7f69148be640 1 -- 172.21.15.82:0/2781774936 wait complete. 2024-06-06T07:36:18.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:18 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2781774936' entity='client.admin' 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.637+0000 7f0a49366640 1 Processor -- start 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.637+0000 7f0a49366640 1 -- start start 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.637+0000 7f0a49366640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a44104350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.637+0000 7f0a49366640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44104920 con 0x7f0a44103f30 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.637+0000 7f0a3affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a44104350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.637+0000 7f0a3affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a44104350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52658/0 (socket says 172.21.15.82:52658) 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.637+0000 7f0a3affd640 1 -- 172.21.15.82:0/144047508 learned_addr learned my addr 172.21.15.82:0/144047508 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.637+0000 7f0a3affd640 1 -- 172.21.15.82:0/144047508 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a44106ec0 con 0x7f0a44103f30 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.638+0000 7f0a3affd640 1 --2- 172.21.15.82:0/144047508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a44104350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009820 tx=0x7f0a3402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be32e68906b18ee3 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:19.393 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.638+0000 7f0a427fc640 1 -- 172.21.15.82:0/144047508 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a3402ad40 con 0x7f0a44103f30 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.638+0000 7f0a427fc640 1 -- 172.21.15.82:0/144047508 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a34032900 con 0x7f0a44103f30 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.638+0000 7f0a427fc640 1 -- 172.21.15.82:0/144047508 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a34030860 con 0x7f0a44103f30 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 -- 172.21.15.82:0/144047508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 msgr2=0x7f0a44104350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 --2- 172.21.15.82:0/144047508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a44104350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009820 tx=0x7f0a3402a270 comp rx=0 tx=0).stop 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 -- 172.21.15.82:0/144047508 shutdown_connections 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 --2- 172.21.15.82:0/144047508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a44104350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 -- 172.21.15.82:0/144047508 >> 172.21.15.82:0/144047508 conn(0x7f0a440ff860 msgr2=0x7f0a44101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 -- 172.21.15.82:0/144047508 shutdown_connections 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 -- 172.21.15.82:0/144047508 wait complete. 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 Processor -- start 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 -- start start 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.639+0000 7f0a49366640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a441066f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:19.394 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a49366640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44106c30 con 0x7f0a44103f30 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a3affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a441066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a3affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a441066f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52666/0 (socket says 172.21.15.82:52666) 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a3affd640 1 -- 172.21.15.82:0/343811483 learned_addr learned my addr 172.21.15.82:0/343811483 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a3affd640 1 -- 172.21.15.82:0/343811483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a340094d0 con 0x7f0a44103f30 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a3affd640 1 --2- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a441066f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0a340097f0 tx=0x7f0a34030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a40ff9640 1 -- 172.21.15.82:0/343811483 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a34040070 con 0x7f0a44103f30 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a40ff9640 1 -- 172.21.15.82:0/343811483 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a34032dd0 con 0x7f0a44103f30 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a44104e70 con 0x7f0a44103f30 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a40ff9640 1 -- 172.21.15.82:0/343811483 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a3403a800 con 0x7f0a44103f30 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.640+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a44105310 con 0x7f0a44103f30 2024-06-06T07:36:19.395 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.641+0000 7f0a40ff9640 1 -- 172.21.15.82:0/343811483 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f0a3403a960 con 0x7f0a44103f30 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.641+0000 7f0a40ff9640 1 --2- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0a300378b0 0x7f0a30039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.641+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4410a0f0 con 0x7f0a44103f30 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.641+0000 7f0a42ffd640 1 --2- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0a300378b0 0x7f0a30039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.641+0000 7f0a40ff9640 1 -- 172.21.15.82:0/343811483 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0a3406a4a0 con 0x7f0a44103f30 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.642+0000 7f0a42ffd640 1 --2- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0a300378b0 0x7f0a30039d70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0098c0 tx=0x7f0a2c006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.646+0000 7f0a40ff9640 1 -- 172.21.15.82:0/343811483 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0a34039780 con 0x7f0a44103f30 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.818+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f0a44105800 con 0x7f0a44103f30 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.821+0000 7f0a40ff9640 1 -- 172.21.15.82:0/343811483 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0a34031ed0 con 0x7f0a44103f30 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0a300378b0 msgr2=0x7f0a30039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 --2- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0a300378b0 0x7f0a30039d70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0098c0 tx=0x7f0a2c006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 msgr2=0x7f0a441066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 --2- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a441066f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0a340097f0 tx=0x7f0a34030e90 comp rx=0 tx=0).stop 2024-06-06T07:36:19.396 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 shutdown_connections 2024-06-06T07:36:19.397 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 --2- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0a300378b0 0x7f0a30039d70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:19.397 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 --2- 172.21.15.82:0/343811483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a44103f30 0x7f0a441066f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:19.397 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 >> 172.21.15.82:0/343811483 conn(0x7f0a440ff860 msgr2=0x7f0a441001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:19.397 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 shutdown_connections 2024-06-06T07:36:19.397 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:18.824+0000 7f0a49366640 1 -- 172.21.15.82:0/343811483 wait complete. 2024-06-06T07:36:19.397 INFO:teuthology.orchestra.run.smithi082.stdout:Enabling the dashboard module... 2024-06-06T07:36:19.641 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:19 smithi082 ceph-mon[29354]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:19.642 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:19 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/343811483' entity='client.admin' 2024-06-06T07:36:20.725 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:20 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/4034980480' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-06T07:36:21.040 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.833+0000 7f092dc31640 1 Processor -- start 2024-06-06T07:36:21.040 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.833+0000 7f092dc31640 1 -- start start 2024-06-06T07:36:21.040 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.833+0000 7f092dc31640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f0928106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:21.040 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.833+0000 7f092dc31640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0928106630 con 0x7f0928105c40 2024-06-06T07:36:21.040 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.833+0000 7f09277fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f0928106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:21.040 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.833+0000 7f09277fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f0928106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52672/0 (socket says 172.21.15.82:52672) 2024-06-06T07:36:21.040 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.833+0000 7f09277fe640 1 -- 172.21.15.82:0/3656740814 learned_addr learned my addr 172.21.15.82:0/3656740814 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:21.040 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.833+0000 7f09277fe640 1 -- 172.21.15.82:0/3656740814 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0928106e40 con 0x7f0928105c40 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.834+0000 7f09277fe640 1 --2- 172.21.15.82:0/3656740814 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f0928106060 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0918009820 tx=0x7f091802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809a799138dea65f server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.834+0000 7f09267fc640 1 -- 172.21.15.82:0/3656740814 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f091802ad40 con 0x7f0928105c40 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.834+0000 7f09267fc640 1 -- 172.21.15.82:0/3656740814 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0918032900 con 0x7f0928105c40 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.834+0000 7f09267fc640 1 -- 172.21.15.82:0/3656740814 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0918030860 con 0x7f0928105c40 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.835+0000 7f092dc31640 1 -- 172.21.15.82:0/3656740814 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 msgr2=0x7f0928106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.835+0000 7f092dc31640 1 --2- 172.21.15.82:0/3656740814 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f0928106060 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0918009820 tx=0x7f091802a270 comp rx=0 tx=0).stop 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.835+0000 7f092dc31640 1 -- 172.21.15.82:0/3656740814 shutdown_connections 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.835+0000 7f092dc31640 1 --2- 172.21.15.82:0/3656740814 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f0928106060 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.835+0000 7f092dc31640 1 -- 172.21.15.82:0/3656740814 >> 172.21.15.82:0/3656740814 conn(0x7f09281011c0 msgr2=0x7f0928103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.835+0000 7f092dc31640 1 -- 172.21.15.82:0/3656740814 shutdown_connections 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.835+0000 7f092dc31640 1 -- 172.21.15.82:0/3656740814 wait complete. 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.835+0000 7f092dc31640 1 Processor -- start 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f092dc31640 1 -- start start 2024-06-06T07:36:21.041 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f092dc31640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f092807bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f092dc31640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092807bff0 con 0x7f0928105c40 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f09277fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f092807bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f09277fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f092807bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52678/0 (socket says 172.21.15.82:52678) 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f09277fe640 1 -- 172.21.15.82:0/4034980480 learned_addr learned my addr 172.21.15.82:0/4034980480 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f09277fe640 1 -- 172.21.15.82:0/4034980480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09180094d0 con 0x7f0928105c40 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f09277fe640 1 --2- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f092807bab0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09180097f0 tx=0x7f0918030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f0924ff9640 1 -- 172.21.15.82:0/4034980480 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0918040070 con 0x7f0928105c40 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f0924ff9640 1 -- 172.21.15.82:0/4034980480 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0918032dd0 con 0x7f0928105c40 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092807a230 con 0x7f0928105c40 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f0924ff9640 1 -- 172.21.15.82:0/4034980480 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f091803a800 con 0x7f0928105c40 2024-06-06T07:36:21.042 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.836+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092807a6d0 con 0x7f0928105c40 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.838+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092810a070 con 0x7f0928105c40 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.838+0000 7f0924ff9640 1 -- 172.21.15.82:0/4034980480 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f091803ccc0 con 0x7f0928105c40 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.838+0000 7f0924ff9640 1 --2- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0910037860 0x7f0910039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.838+0000 7f0924ff9640 1 -- 172.21.15.82:0/4034980480 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f091806a5e0 con 0x7f0928105c40 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.838+0000 7f0926ffd640 1 --2- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0910037860 0x7f0910039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.839+0000 7f0926ffd640 1 --2- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0910037860 0x7f0910039d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f091c0098c0 tx=0x7f091c006db0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:19.843+0000 7f0924ff9640 1 -- 172.21.15.82:0/4034980480 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0918043b00 con 0x7f0928105c40 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.061+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f092810a2b0 con 0x7f0928105c40 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.427+0000 7f0924ff9640 1 -- 172.21.15.82:0/4034980480 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 9) v1 ==== 44879+0+0 (secure 0 0 0) 0x7f091803bdc0 con 0x7f0928105c40 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.428+0000 7f0924ff9640 1 -- 172.21.15.82:0/4034980480 <== mon.0 v2:172.21.15.82:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0918069f30 con 0x7f0928105c40 2024-06-06T07:36:21.043 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0910037860 msgr2=0x7f0910039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:21.044 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 --2- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0910037860 0x7f0910039d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f091c0098c0 tx=0x7f091c006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:21.044 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 msgr2=0x7f092807bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:21.044 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 --2- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f092807bab0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09180097f0 tx=0x7f0918030e90 comp rx=0 tx=0).stop 2024-06-06T07:36:21.044 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 shutdown_connections 2024-06-06T07:36:21.044 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 --2- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f0910037860 0x7f0910039d20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:21.044 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 --2- 172.21.15.82:0/4034980480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0928105c40 0x7f092807bab0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:21.044 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 >> 172.21.15.82:0/4034980480 conn(0x7f09281011c0 msgr2=0x7f0928101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:21.044 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 shutdown_connections 2024-06-06T07:36:21.044 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:20.430+0000 7f092dc31640 1 -- 172.21.15.82:0/4034980480 wait complete. 2024-06-06T07:36:21.595 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:21 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/4034980480' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-06T07:36:21.595 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:21 smithi082 ceph-mon[29354]: mgrmap e9: smithi082.kocfig(active, since 25s) 2024-06-06T07:36:22.239 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout { 2024-06-06T07:36:22.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-06T07:36:22.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-06T07:36:22.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "active_name": "smithi082.kocfig", 2024-06-06T07:36:22.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-06T07:36:22.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout } 2024-06-06T07:36:22.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.389+0000 7f38cf9e7640 1 Processor -- start 2024-06-06T07:36:22.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.389+0000 7f38cf9e7640 1 -- start start 2024-06-06T07:36:22.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.389+0000 7f38cf9e7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c8108a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:22.240 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.389+0000 7f38cf9e7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c8079e40 con 0x7f38c8106680 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.390+0000 7f38cd783640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c8108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.390+0000 7f38cd783640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c8108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52696/0 (socket says 172.21.15.82:52696) 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.390+0000 7f38cd783640 1 -- 172.21.15.82:0/1764509231 learned_addr learned my addr 172.21.15.82:0/1764509231 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.390+0000 7f38cd783640 1 -- 172.21.15.82:0/1764509231 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c8108fb0 con 0x7f38c8106680 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.391+0000 7f38cd783640 1 --2- 172.21.15.82:0/1764509231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c8108a70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009820 tx=0x7f38c402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bfb03b429921a6 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.391+0000 7f38bbfff640 1 -- 172.21.15.82:0/1764509231 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c402ad40 con 0x7f38c8106680 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.391+0000 7f38bbfff640 1 -- 172.21.15.82:0/1764509231 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38c4032900 con 0x7f38c8106680 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.391+0000 7f38bbfff640 1 -- 172.21.15.82:0/1764509231 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c4030860 con 0x7f38c8106680 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.391+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1764509231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 msgr2=0x7f38c8108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.391+0000 7f38cf9e7640 1 --2- 172.21.15.82:0/1764509231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c8108a70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009820 tx=0x7f38c402a270 comp rx=0 tx=0).stop 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1764509231 shutdown_connections 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cf9e7640 1 --2- 172.21.15.82:0/1764509231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c8108a70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:22.241 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1764509231 >> 172.21.15.82:0/1764509231 conn(0x7f38c81000d0 msgr2=0x7f38c81024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1764509231 shutdown_connections 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1764509231 wait complete. 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cf9e7640 1 Processor -- start 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cf9e7640 1 -- start start 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cf9e7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c81998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cf9e7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c8199df0 con 0x7f38c8106680 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cd783640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c81998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.392+0000 7f38cd783640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c81998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52704/0 (socket says 172.21.15.82:52704) 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.393+0000 7f38cd783640 1 -- 172.21.15.82:0/1490797687 learned_addr learned my addr 172.21.15.82:0/1490797687 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.393+0000 7f38cd783640 1 -- 172.21.15.82:0/1490797687 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c40094d0 con 0x7f38c8106680 2024-06-06T07:36:22.242 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.393+0000 7f38cd783640 1 --2- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c81998b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f38c4032670 tx=0x7f38c40326a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.393+0000 7f38ba7fc640 1 -- 172.21.15.82:0/1490797687 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c4032c80 con 0x7f38c8106680 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.393+0000 7f38ba7fc640 1 -- 172.21.15.82:0/1490797687 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38c4032de0 con 0x7f38c8106680 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.393+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c8199ff0 con 0x7f38c8106680 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.393+0000 7f38ba7fc640 1 -- 172.21.15.82:0/1490797687 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c403a7c0 con 0x7f38c8106680 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.393+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c819a490 con 0x7f38c8106680 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.394+0000 7f38ba7fc640 1 -- 172.21.15.82:0/1490797687 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 9) v1 ==== 44879+0+0 (secure 0 0 0) 0x7f38c403a9b0 con 0x7f38c8106680 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.394+0000 7f38ba7fc640 1 --2- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f38b4037900 0x7f38b4039dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.394+0000 7f38ba7fc640 1 -- 172.21.15.82:0/1490797687 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f38c406ba20 con 0x7f38c8106680 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.394+0000 7f38ccf82640 1 -- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f38b4037900 msgr2=0x7f38b4039dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1662212559 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.394+0000 7f38ccf82640 1 --2- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f38b4037900 0x7f38b4039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.394+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c8061ae0 con 0x7f38c8106680 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.399+0000 7f38ba7fc640 1 -- 172.21.15.82:0/1490797687 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f38c4039d50 con 0x7f38c8106680 2024-06-06T07:36:22.243 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.595+0000 7f38ccf82640 1 -- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f38b4037900 msgr2=0x7f38b4039dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1662212559 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.595+0000 7f38ccf82640 1 --2- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f38b4037900 0x7f38b4039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.628+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f38c819cf70 con 0x7f38c8106680 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.629+0000 7f38ba7fc640 1 -- 172.21.15.82:0/1490797687 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f38c4039310 con 0x7f38c8106680 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.630+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f38b4037900 msgr2=0x7f38b4039dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.630+0000 7f38cf9e7640 1 --2- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f38b4037900 0x7f38b4039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.630+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 msgr2=0x7f38c81998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.630+0000 7f38cf9e7640 1 --2- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c81998b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f38c4032670 tx=0x7f38c40326a0 comp rx=0 tx=0).stop 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.630+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 shutdown_connections 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.630+0000 7f38cf9e7640 1 --2- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7f38b4037900 0x7f38b4039dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.630+0000 7f38cf9e7640 1 --2- 172.21.15.82:0/1490797687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38c8106680 0x7f38c81998b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.630+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 >> 172.21.15.82:0/1490797687 conn(0x7f38c81000d0 msgr2=0x7f38c8100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:22.244 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.631+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 shutdown_connections 2024-06-06T07:36:22.245 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:21.631+0000 7f38cf9e7640 1 -- 172.21.15.82:0/1490797687 wait complete. 2024-06-06T07:36:22.245 INFO:teuthology.orchestra.run.smithi082.stdout:Waiting for the mgr to restart... 2024-06-06T07:36:22.245 INFO:teuthology.orchestra.run.smithi082.stdout:Waiting for mgr epoch 9... 2024-06-06T07:36:22.616 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:22 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1490797687' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-06T07:36:25.497 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: Active manager daemon smithi082.kocfig restarted 2024-06-06T07:36:25.497 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: Activating manager daemon smithi082.kocfig 2024-06-06T07:36:25.497 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: osdmap e3: 0 total, 0 up, 0 in 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: mgrmap e10: smithi082.kocfig(active, starting, since 0.00549521s) 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr metadata", "who": "smithi082.kocfig", "id": "smithi082.kocfig"}]: dispatch 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: Manager daemon smithi082.kocfig is now available 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi082.kocfig/mirror_snapshot_schedule"}]: dispatch 2024-06-06T07:36:25.498 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:25 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi082.kocfig/trash_purge_schedule"}]: dispatch 2024-06-06T07:36:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout { 2024-06-06T07:36:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-06T07:36:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-06T07:36:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout } 2024-06-06T07:36:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.679+0000 7fe47a20a640 1 Processor -- start 2024-06-06T07:36:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.679+0000 7fe47a20a640 1 -- start start 2024-06-06T07:36:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.679+0000 7fe47a20a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe474108a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.679+0000 7fe47a20a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe474079e40 con 0x7fe474106680 2024-06-06T07:36:27.066 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.680+0000 7fe4737fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe474108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.680+0000 7fe4737fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe474108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52718/0 (socket says 172.21.15.82:52718) 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.680+0000 7fe4737fe640 1 -- 172.21.15.82:0/3874751082 learned_addr learned my addr 172.21.15.82:0/3874751082 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.680+0000 7fe4737fe640 1 -- 172.21.15.82:0/3874751082 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe474108fb0 con 0x7fe474106680 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe4737fe640 1 --2- 172.21.15.82:0/3874751082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe474108a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009820 tx=0x7fe45c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04d11b9865c61dd server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe4727fc640 1 -- 172.21.15.82:0/3874751082 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe45c02ad40 con 0x7fe474106680 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe4727fc640 1 -- 172.21.15.82:0/3874751082 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe45c032900 con 0x7fe474106680 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe4727fc640 1 -- 172.21.15.82:0/3874751082 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe45c030860 con 0x7fe474106680 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe47a20a640 1 -- 172.21.15.82:0/3874751082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 msgr2=0x7fe474108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe47a20a640 1 --2- 172.21.15.82:0/3874751082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe474108a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009820 tx=0x7fe45c02a270 comp rx=0 tx=0).stop 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe47a20a640 1 -- 172.21.15.82:0/3874751082 shutdown_connections 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe47a20a640 1 --2- 172.21.15.82:0/3874751082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe474108a70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe47a20a640 1 -- 172.21.15.82:0/3874751082 >> 172.21.15.82:0/3874751082 conn(0x7fe4741000d0 msgr2=0x7fe4741024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:27.067 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe47a20a640 1 -- 172.21.15.82:0/3874751082 shutdown_connections 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.681+0000 7fe47a20a640 1 -- 172.21.15.82:0/3874751082 wait complete. 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.682+0000 7fe47a20a640 1 Processor -- start 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.682+0000 7fe47a20a640 1 -- start start 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.682+0000 7fe47a20a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe4741998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.682+0000 7fe47a20a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe474199df0 con 0x7fe474106680 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.682+0000 7fe4737fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe4741998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.682+0000 7fe4737fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe4741998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52726/0 (socket says 172.21.15.82:52726) 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.682+0000 7fe4737fe640 1 -- 172.21.15.82:0/3115840634 learned_addr learned my addr 172.21.15.82:0/3115840634 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.683+0000 7fe4737fe640 1 -- 172.21.15.82:0/3115840634 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c0094d0 con 0x7fe474106680 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.683+0000 7fe4737fe640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe4741998b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0097f0 tx=0x7fe45c030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.683+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe45c040070 con 0x7fe474106680 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.683+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe45c032dd0 con 0x7fe474106680 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.683+0000 7fe47a20a640 1 -- 172.21.15.82:0/3115840634 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe474199ff0 con 0x7fe474106680 2024-06-06T07:36:27.068 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.683+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe45c03a800 con 0x7fe474106680 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.683+0000 7fe47a20a640 1 -- 172.21.15.82:0/3115840634 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47419a490 con 0x7fe474106680 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.684+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 9) v1 ==== 44879+0+0 (secure 0 0 0) 0x7fe45c03a960 con 0x7fe474106680 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.684+0000 7fe470ff9640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 0x7fe460039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.684+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 --> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe46003a480 con 0x7fe4600378b0 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.684+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe45c06a6a0 con 0x7fe474106680 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.684+0000 7fe472ffd640 1 -- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 msgr2=0x7fe460039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1662212559 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.684+0000 7fe472ffd640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 0x7fe460039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.885+0000 7fe472ffd640 1 -- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 msgr2=0x7fe460039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1662212559 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:22.885+0000 7fe472ffd640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 0x7fe460039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:23.286+0000 7fe472ffd640 1 -- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 msgr2=0x7fe460039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1662212559 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:23.286+0000 7fe472ffd640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 0x7fe460039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:24.087+0000 7fe472ffd640 1 -- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 msgr2=0x7fe460039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/1662212559 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:24.087+0000 7fe472ffd640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 0x7fe460039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-06T07:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:25.354+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mgrmap(e 10) v1 ==== 44646+0+0 (secure 0 0 0) 0x7fe45c03ccc0 con 0x7fe474106680 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:25.354+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 msgr2=0x7fe460039d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:25.354+0000 7fe470ff9640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 0x7fe460039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.353+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 11) v1 ==== 44773+0+0 (secure 0 0 0) 0x7fe45c03ac10 con 0x7fe474106680 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.353+0000 7fe470ff9640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe46003b370 0x7fe46003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.353+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 --> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe46003a480 con 0x7fe46003b370 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.354+0000 7fe472ffd640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe46003b370 0x7fe46003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.354+0000 7fe472ffd640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe46003b370 0x7fe46003d760 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe464003980 tx=0x7fe464006e10 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.355+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 <== mgr.14160 v2:172.21.15.82:6800/493557010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fe46003a480 con 0x7fe46003b370 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 -- 172.21.15.82:0/3115840634 --> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe474109e10 con 0x7fe46003b370 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe470ff9640 1 -- 172.21.15.82:0/3115840634 <== mgr.14160 v2:172.21.15.82:6800/493557010 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fe474109e10 con 0x7fe46003b370 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 -- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe46003b370 msgr2=0x7fe46003d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe46003b370 0x7fe46003d760 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe464003980 tx=0x7fe464006e10 comp rx=0 tx=0).stop 2024-06-06T07:36:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 -- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 msgr2=0x7fe4741998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe4741998b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0097f0 tx=0x7fe45c030e90 comp rx=0 tx=0).stop 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 -- 172.21.15.82:0/3115840634 shutdown_connections 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe46003b370 0x7fe46003d760 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:6800/1662212559,v1:172.21.15.82:6801/1662212559] conn(0x7fe4600378b0 0x7fe460039d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 --2- 172.21.15.82:0/3115840634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe474106680 0x7fe4741998b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 -- 172.21.15.82:0/3115840634 >> 172.21.15.82:0/3115840634 conn(0x7fe4741000d0 msgr2=0x7fe474100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 -- 172.21.15.82:0/3115840634 shutdown_connections 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:26.361+0000 7fe47a20a640 1 -- 172.21.15.82:0/3115840634 wait complete. 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:mgr epoch 9 is available 2024-06-06T07:36:27.071 INFO:teuthology.orchestra.run.smithi082.stdout:Generating a dashboard self-signed certificate... 2024-06-06T07:36:27.734 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:27 smithi082 ceph-mon[29354]: mgrmap e11: smithi082.kocfig(active, since 1.00526s) 2024-06-06T07:36:27.734 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:27 smithi082 ceph-mon[29354]: [06/Jun/2024:07:36:26] ENGINE Bus STARTING 2024-06-06T07:36:27.734 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:27 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:28.387 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-06T07:36:28.387 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.524+0000 7efdd60ed640 1 Processor -- start 2024-06-06T07:36:28.387 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.524+0000 7efdd60ed640 1 -- start start 2024-06-06T07:36:28.387 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.524+0000 7efdd60ed640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd0104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:28.387 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.524+0000 7efdd60ed640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0105500 con 0x7efdd0104b10 2024-06-06T07:36:28.387 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.524+0000 7efdcf7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd0104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:28.387 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.524+0000 7efdcf7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd0104f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56876/0 (socket says 172.21.15.82:56876) 2024-06-06T07:36:28.387 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.524+0000 7efdcf7fe640 1 -- 172.21.15.82:0/4151072911 learned_addr learned my addr 172.21.15.82:0/4151072911 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.524+0000 7efdcf7fe640 1 -- 172.21.15.82:0/4151072911 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd0105d10 con 0x7efdd0104b10 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.525+0000 7efdcf7fe640 1 --2- 172.21.15.82:0/4151072911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd0104f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009820 tx=0x7efdc002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582aac38d2785483 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.525+0000 7efdce7fc640 1 -- 172.21.15.82:0/4151072911 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc002ad40 con 0x7efdd0104b10 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.525+0000 7efdce7fc640 1 -- 172.21.15.82:0/4151072911 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdc0032900 con 0x7efdd0104b10 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.525+0000 7efdce7fc640 1 -- 172.21.15.82:0/4151072911 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc0030860 con 0x7efdd0104b10 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.526+0000 7efdd60ed640 1 -- 172.21.15.82:0/4151072911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 msgr2=0x7efdd0104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.526+0000 7efdd60ed640 1 --2- 172.21.15.82:0/4151072911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd0104f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009820 tx=0x7efdc002a270 comp rx=0 tx=0).stop 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.526+0000 7efdd60ed640 1 -- 172.21.15.82:0/4151072911 shutdown_connections 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.526+0000 7efdd60ed640 1 --2- 172.21.15.82:0/4151072911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd0104f30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.526+0000 7efdd60ed640 1 -- 172.21.15.82:0/4151072911 >> 172.21.15.82:0/4151072911 conn(0x7efdd01000d0 msgr2=0x7efdd01024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.526+0000 7efdd60ed640 1 -- 172.21.15.82:0/4151072911 shutdown_connections 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.526+0000 7efdd60ed640 1 -- 172.21.15.82:0/4151072911 wait complete. 2024-06-06T07:36:28.388 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.527+0000 7efdd60ed640 1 Processor -- start 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.527+0000 7efdd60ed640 1 -- start start 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.527+0000 7efdd60ed640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd01998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.527+0000 7efdd60ed640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0199e30 con 0x7efdd0104b10 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.527+0000 7efdcf7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd01998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.527+0000 7efdcf7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd01998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56882/0 (socket says 172.21.15.82:56882) 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.527+0000 7efdcf7fe640 1 -- 172.21.15.82:0/79778278 learned_addr learned my addr 172.21.15.82:0/79778278 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.527+0000 7efdcf7fe640 1 -- 172.21.15.82:0/79778278 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00094d0 con 0x7efdd0104b10 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.527+0000 7efdcf7fe640 1 --2- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd01998f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009940 tx=0x7efdc0032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.528+0000 7efdccff9640 1 -- 172.21.15.82:0/79778278 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc0032c80 con 0x7efdd0104b10 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.528+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd019a030 con 0x7efdd0104b10 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.528+0000 7efdccff9640 1 -- 172.21.15.82:0/79778278 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdc0032de0 con 0x7efdd0104b10 2024-06-06T07:36:28.389 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.528+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd019a4d0 con 0x7efdd0104b10 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.528+0000 7efdccff9640 1 -- 172.21.15.82:0/79778278 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc003a730 con 0x7efdd0104b10 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.528+0000 7efdccff9640 1 -- 172.21.15.82:0/79778278 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 11) v1 ==== 44773+0+0 (secure 0 0 0) 0x7efdc003a9f0 con 0x7efdd0104b10 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.529+0000 7efdccff9640 1 --2- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7efdb8037790 0x7efdb8039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.529+0000 7efdccff9640 1 -- 172.21.15.82:0/79778278 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efdc006b950 con 0x7efdd0104b10 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.529+0000 7efdceffd640 1 --2- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7efdb8037790 0x7efdb8039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.529+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd0108f40 con 0x7efdd0104b10 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.529+0000 7efdceffd640 1 --2- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7efdb8037790 0x7efdb8039c50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efdc40098c0 tx=0x7efdc4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.534+0000 7efdccff9640 1 -- 172.21.15.82:0/79778278 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7efdc0039d30 con 0x7efdd0104b10 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.733+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 --> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7efdd0101240 con 0x7efdb8037790 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.789+0000 7efdccff9640 1 -- 172.21.15.82:0/79778278 <== mgr.14160 v2:172.21.15.82:6800/493557010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7efdd0101240 con 0x7efdb8037790 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7efdb8037790 msgr2=0x7efdb8039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 --2- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7efdb8037790 0x7efdb8039c50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efdc40098c0 tx=0x7efdc4006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:28.390 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 msgr2=0x7efdd01998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:28.391 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 --2- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd01998f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009940 tx=0x7efdc0032670 comp rx=0 tx=0).stop 2024-06-06T07:36:28.391 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 shutdown_connections 2024-06-06T07:36:28.391 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 --2- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7efdb8037790 0x7efdb8039c50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:28.391 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 --2- 172.21.15.82:0/79778278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdd0104b10 0x7efdd01998f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:28.391 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 >> 172.21.15.82:0/79778278 conn(0x7efdd01000d0 msgr2=0x7efdd0100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:28.391 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 shutdown_connections 2024-06-06T07:36:28.391 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:27.791+0000 7efdd60ed640 1 -- 172.21.15.82:0/79778278 wait complete. 2024-06-06T07:36:28.391 INFO:teuthology.orchestra.run.smithi082.stdout:Creating initial admin user... 2024-06-06T07:36:28.507 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:28 smithi082 ceph-mon[29354]: [06/Jun/2024:07:36:26] ENGINE Serving on http://172.21.15.82:8765 2024-06-06T07:36:28.507 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:28 smithi082 ceph-mon[29354]: [06/Jun/2024:07:36:26] ENGINE Serving on https://172.21.15.82:7150 2024-06-06T07:36:28.507 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:28 smithi082 ceph-mon[29354]: [06/Jun/2024:07:36:26] ENGINE Bus STARTED 2024-06-06T07:36:28.507 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:28 smithi082 ceph-mon[29354]: [06/Jun/2024:07:36:26] ENGINE Error in HTTPServer.serve 2024-06-06T07:36:28.507 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Traceback (most recent call last): 2024-06-06T07:36:28.507 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-06T07:36:28.507 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self._connections.run(self.expiration_interval) 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self._run(expiration_interval) 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: s = self.context.wrap_socket( 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: return self.sslsocket_class._create( 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self.do_handshake() 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-06T07:36:28.508 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self._sslobj.do_handshake() 2024-06-06T07:36:28.509 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-06T07:36:28.509 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:28 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:28.509 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:28 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:28.509 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:28 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:28.509 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:28 smithi082 ceph-mon[29354]: mgrmap e12: smithi082.kocfig(active, since 2s) 2024-06-06T07:36:29.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:29 smithi082 ceph-mon[29354]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:29.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:29 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:30.192 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$kRUIVglXytzuFwBRgM44RuCDVEV87qSlGkDXh6xYXIXaUPArGzZei", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1717659389, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-06T07:36:30.192 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.805+0000 7f4b62309640 1 Processor -- start 2024-06-06T07:36:30.193 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.805+0000 7f4b62309640 1 -- start start 2024-06-06T07:36:30.193 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.805+0000 7f4b62309640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:30.193 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.805+0000 7f4b62309640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c106630 con 0x7f4b5c105c40 2024-06-06T07:36:30.193 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.805+0000 7f4b5bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:30.193 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.805+0000 7f4b5bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56898/0 (socket says 172.21.15.82:56898) 2024-06-06T07:36:30.194 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.805+0000 7f4b5bfff640 1 -- 172.21.15.82:0/2706928731 learned_addr learned my addr 172.21.15.82:0/2706928731 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:30.194 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.806+0000 7f4b5bfff640 1 -- 172.21.15.82:0/2706928731 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c106e40 con 0x7f4b5c105c40 2024-06-06T07:36:30.194 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.806+0000 7f4b5bfff640 1 --2- 172.21.15.82:0/2706928731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c106060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009820 tx=0x7f4b4c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44915278af36cd43 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:30.194 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.806+0000 7f4b5affd640 1 -- 172.21.15.82:0/2706928731 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b4c02ad40 con 0x7f4b5c105c40 2024-06-06T07:36:30.194 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.807+0000 7f4b5affd640 1 -- 172.21.15.82:0/2706928731 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b4c032900 con 0x7f4b5c105c40 2024-06-06T07:36:30.194 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.807+0000 7f4b5affd640 1 -- 172.21.15.82:0/2706928731 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b4c030860 con 0x7f4b5c105c40 2024-06-06T07:36:30.194 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.807+0000 7f4b62309640 1 -- 172.21.15.82:0/2706928731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 msgr2=0x7f4b5c106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:30.194 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.807+0000 7f4b62309640 1 --2- 172.21.15.82:0/2706928731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c106060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009820 tx=0x7f4b4c02a270 comp rx=0 tx=0).stop 2024-06-06T07:36:30.195 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.807+0000 7f4b62309640 1 -- 172.21.15.82:0/2706928731 shutdown_connections 2024-06-06T07:36:30.195 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.807+0000 7f4b62309640 1 --2- 172.21.15.82:0/2706928731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c106060 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:30.195 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.807+0000 7f4b62309640 1 -- 172.21.15.82:0/2706928731 >> 172.21.15.82:0/2706928731 conn(0x7f4b5c1011c0 msgr2=0x7f4b5c103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:30.195 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.807+0000 7f4b62309640 1 -- 172.21.15.82:0/2706928731 shutdown_connections 2024-06-06T07:36:30.195 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.807+0000 7f4b62309640 1 -- 172.21.15.82:0/2706928731 wait complete. 2024-06-06T07:36:30.195 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.808+0000 7f4b62309640 1 Processor -- start 2024-06-06T07:36:30.195 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.808+0000 7f4b62309640 1 -- start start 2024-06-06T07:36:30.196 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.808+0000 7f4b62309640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:30.196 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.808+0000 7f4b62309640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c199df0 con 0x7f4b5c105c40 2024-06-06T07:36:30.196 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.808+0000 7f4b5bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:30.196 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.808+0000 7f4b5bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c1998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56912/0 (socket says 172.21.15.82:56912) 2024-06-06T07:36:30.196 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.808+0000 7f4b5bfff640 1 -- 172.21.15.82:0/2766382518 learned_addr learned my addr 172.21.15.82:0/2766382518 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:30.196 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.808+0000 7f4b5bfff640 1 -- 172.21.15.82:0/2766382518 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c0094d0 con 0x7f4b5c105c40 2024-06-06T07:36:30.196 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.808+0000 7f4b5bfff640 1 --2- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c1998b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009940 tx=0x7f4b4c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:30.196 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.809+0000 7f4b597fa640 1 -- 172.21.15.82:0/2766382518 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b4c032c80 con 0x7f4b5c105c40 2024-06-06T07:36:30.197 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.809+0000 7f4b597fa640 1 -- 172.21.15.82:0/2766382518 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b4c032de0 con 0x7f4b5c105c40 2024-06-06T07:36:30.197 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.809+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c199ff0 con 0x7f4b5c105c40 2024-06-06T07:36:30.197 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.809+0000 7f4b597fa640 1 -- 172.21.15.82:0/2766382518 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b4c03a730 con 0x7f4b5c105c40 2024-06-06T07:36:30.197 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.809+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c19a490 con 0x7f4b5c105c40 2024-06-06T07:36:30.197 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.810+0000 7f4b597fa640 1 -- 172.21.15.82:0/2766382518 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 12) v1 ==== 44879+0+0 (secure 0 0 0) 0x7f4b4c03a9f0 con 0x7f4b5c105c40 2024-06-06T07:36:30.197 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.810+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c10a070 con 0x7f4b5c105c40 2024-06-06T07:36:30.197 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.810+0000 7f4b597fa640 1 --2- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f4b440378b0 0x7f4b44039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:30.197 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.810+0000 7f4b597fa640 1 -- 172.21.15.82:0/2766382518 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4b4c06ac70 con 0x7f4b5c105c40 2024-06-06T07:36:30.197 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.810+0000 7f4b5b7fe640 1 --2- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f4b440378b0 0x7f4b44039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:30.198 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.810+0000 7f4b5b7fe640 1 --2- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f4b440378b0 0x7f4b44039d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b500098c0 tx=0x7f4b50006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:30.198 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:28.814+0000 7f4b597fa640 1 -- 172.21.15.82:0/2766382518 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4b4c039b60 con 0x7f4b5c105c40 2024-06-06T07:36:30.198 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.007+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 --> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4b5c10a2b0 con 0x7f4b440378b0 2024-06-06T07:36:30.198 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.232+0000 7f4b597fa640 1 -- 172.21.15.82:0/2766382518 <== mgr.14160 v2:172.21.15.82:6800/493557010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f4b5c10a2b0 con 0x7f4b440378b0 2024-06-06T07:36:30.198 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f4b440378b0 msgr2=0x7f4b44039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:30.198 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 --2- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f4b440378b0 0x7f4b44039d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b500098c0 tx=0x7f4b50006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:30.198 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 msgr2=0x7f4b5c1998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:30.198 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 --2- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c1998b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009940 tx=0x7f4b4c032670 comp rx=0 tx=0).stop 2024-06-06T07:36:30.198 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 shutdown_connections 2024-06-06T07:36:30.199 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 --2- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f4b440378b0 0x7f4b44039d70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:30.199 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 --2- 172.21.15.82:0/2766382518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b5c105c40 0x7f4b5c1998b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:30.199 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 >> 172.21.15.82:0/2766382518 conn(0x7f4b5c1011c0 msgr2=0x7f4b5c101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:30.199 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 shutdown_connections 2024-06-06T07:36:30.199 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:29.235+0000 7f4b62309640 1 -- 172.21.15.82:0/2766382518 wait complete. 2024-06-06T07:36:30.199 INFO:teuthology.orchestra.run.smithi082.stdout:Fetching dashboard port number... 2024-06-06T07:36:30.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:30 smithi082 ceph-mon[29354]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:31.497 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:31 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3618884164' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-06T07:36:31.771 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stdout 8443 2024-06-06T07:36:31.771 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.012+0000 7f3002fdb640 1 Processor -- start 2024-06-06T07:36:31.771 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.012+0000 7f3002fdb640 1 -- start start 2024-06-06T07:36:31.771 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.012+0000 7f3002fdb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc108ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:31.771 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.012+0000 7f3002fdb640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc079de0 con 0x7f2ffc1066e0 2024-06-06T07:36:31.771 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.013+0000 7f3000d77640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc108ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:31.771 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.013+0000 7f3000d77640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc108ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56916/0 (socket says 172.21.15.82:56916) 2024-06-06T07:36:31.771 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.013+0000 7f3000d77640 1 -- 172.21.15.82:0/1105746742 learned_addr learned my addr 172.21.15.82:0/1105746742 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.013+0000 7f3000d77640 1 -- 172.21.15.82:0/1105746742 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc109010 con 0x7f2ffc1066e0 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f3000d77640 1 --2- 172.21.15.82:0/1105746742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc108ad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009820 tx=0x7f2fec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4717dbffc27aa5e0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f2ffb7fe640 1 -- 172.21.15.82:0/1105746742 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fec02ad40 con 0x7f2ffc1066e0 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f2ffb7fe640 1 -- 172.21.15.82:0/1105746742 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fec032900 con 0x7f2ffc1066e0 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f2ffb7fe640 1 -- 172.21.15.82:0/1105746742 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fec030860 con 0x7f2ffc1066e0 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f3002fdb640 1 -- 172.21.15.82:0/1105746742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 msgr2=0x7f2ffc108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f3002fdb640 1 --2- 172.21.15.82:0/1105746742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc108ad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009820 tx=0x7f2fec02a270 comp rx=0 tx=0).stop 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f3002fdb640 1 -- 172.21.15.82:0/1105746742 shutdown_connections 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f3002fdb640 1 --2- 172.21.15.82:0/1105746742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc108ad0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f3002fdb640 1 -- 172.21.15.82:0/1105746742 >> 172.21.15.82:0/1105746742 conn(0x7f2ffc1000d0 msgr2=0x7f2ffc1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f3002fdb640 1 -- 172.21.15.82:0/1105746742 shutdown_connections 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.014+0000 7f3002fdb640 1 -- 172.21.15.82:0/1105746742 wait complete. 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.015+0000 7f3002fdb640 1 Processor -- start 2024-06-06T07:36:31.772 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.015+0000 7f3002fdb640 1 -- start start 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.015+0000 7f3002fdb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.015+0000 7f3002fdb640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc199dc0 con 0x7f2ffc1066e0 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.015+0000 7f3000d77640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.015+0000 7f3000d77640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56920/0 (socket says 172.21.15.82:56920) 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.015+0000 7f3000d77640 1 -- 172.21.15.82:0/3618884164 learned_addr learned my addr 172.21.15.82:0/3618884164 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.015+0000 7f3000d77640 1 -- 172.21.15.82:0/3618884164 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec0094d0 con 0x7f2ffc1066e0 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.016+0000 7f3000d77640 1 --2- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc199880 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2fec032670 tx=0x7f2fec0326a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.016+0000 7f2ff9ffb640 1 -- 172.21.15.82:0/3618884164 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fec032c80 con 0x7f2ffc1066e0 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.016+0000 7f2ff9ffb640 1 -- 172.21.15.82:0/3618884164 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fec032de0 con 0x7f2ffc1066e0 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.016+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc199fc0 con 0x7f2ffc1066e0 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.016+0000 7f2ff9ffb640 1 -- 172.21.15.82:0/3618884164 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fec03a7c0 con 0x7f2ffc1066e0 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.016+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc19a460 con 0x7f2ffc1066e0 2024-06-06T07:36:31.773 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.017+0000 7f2ff9ffb640 1 -- 172.21.15.82:0/3618884164 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 12) v1 ==== 44879+0+0 (secure 0 0 0) 0x7f2fec03aa00 con 0x7f2ffc1066e0 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.017+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc109e10 con 0x7f2ffc1066e0 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.017+0000 7f2ff9ffb640 1 --2- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2fe40378b0 0x7f2fe4039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.017+0000 7f2ff9ffb640 1 -- 172.21.15.82:0/3618884164 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2fec06ab80 con 0x7f2ffc1066e0 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.017+0000 7f2ffbfff640 1 --2- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2fe40378b0 0x7f2fe4039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.018+0000 7f2ffbfff640 1 --2- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2fe40378b0 0x7f2fe4039d70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00098c0 tx=0x7f2ff0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.022+0000 7f2ff9ffb640 1 -- 172.21.15.82:0/3618884164 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2fec0394f0 con 0x7f2ffc1066e0 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.192+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2ffc10a050 con 0x7f2ffc1066e0 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.193+0000 7f2ff9ffb640 1 -- 172.21.15.82:0/3618884164 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2fec039310 con 0x7f2ffc1066e0 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2fe40378b0 msgr2=0x7f2fe4039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 --2- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2fe40378b0 0x7f2fe4039d70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00098c0 tx=0x7f2ff0006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 msgr2=0x7f2ffc199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 --2- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc199880 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2fec032670 tx=0x7f2fec0326a0 comp rx=0 tx=0).stop 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 shutdown_connections 2024-06-06T07:36:31.774 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 --2- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2fe40378b0 0x7f2fe4039d70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:31.775 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 --2- 172.21.15.82:0/3618884164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ffc1066e0 0x7f2ffc199880 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:31.775 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 >> 172.21.15.82:0/3618884164 conn(0x7f2ffc1000d0 msgr2=0x7f2ffc100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:31.775 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 shutdown_connections 2024-06-06T07:36:31.775 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:31.195+0000 7f3002fdb640 1 -- 172.21.15.82:0/3618884164 wait complete. 2024-06-06T07:36:31.778 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T07:36:31.778 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout disabled 2024-06-06T07:36:31.785 INFO:teuthology.orchestra.run.smithi082.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T07:36:31.785 INFO:teuthology.orchestra.run.smithi082.stdout:systemctl: stdout inactive 2024-06-06T07:36:31.785 INFO:teuthology.orchestra.run.smithi082.stdout:firewalld.service is not enabled 2024-06-06T07:36:31.785 INFO:teuthology.orchestra.run.smithi082.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-06T07:36:31.786 INFO:teuthology.orchestra.run.smithi082.stdout:Ceph Dashboard is now available at: 2024-06-06T07:36:31.787 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:31.787 INFO:teuthology.orchestra.run.smithi082.stdout: URL: https://smithi082.front.sepia.ceph.com:8443/ 2024-06-06T07:36:31.787 INFO:teuthology.orchestra.run.smithi082.stdout: User: admin 2024-06-06T07:36:31.787 INFO:teuthology.orchestra.run.smithi082.stdout: Password: 46928imvpa 2024-06-06T07:36:31.787 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:31.787 INFO:teuthology.orchestra.run.smithi082.stdout:Saving cluster configuration to /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config directory 2024-06-06T07:36:31.787 INFO:teuthology.orchestra.run.smithi082.stdout:Enabling autotune for osd_memory_target 2024-06-06T07:36:32.687 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:32 smithi082 ceph-mon[29354]: mgrmap e13: smithi082.kocfig(active, since 6s) 2024-06-06T07:36:32.961 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.189+0000 7f36d31a4640 1 Processor -- start 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.189+0000 7f36d31a4640 1 -- start start 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.189+0000 7f36d31a4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc07a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.189+0000 7f36d31a4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc07a840 con 0x7f36cc07be10 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.189+0000 7f36d0f40640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc07a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.189+0000 7f36d0f40640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc07a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56924/0 (socket says 172.21.15.82:56924) 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.189+0000 7f36d0f40640 1 -- 172.21.15.82:0/3083936533 learned_addr learned my addr 172.21.15.82:0/3083936533 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.190+0000 7f36d0f40640 1 -- 172.21.15.82:0/3083936533 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc07a980 con 0x7f36cc07be10 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.190+0000 7f36d0f40640 1 --2- 172.21.15.82:0/3083936533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc07a270 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009820 tx=0x7f36c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90b78fedc3f484e server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.190+0000 7f36cb7fe640 1 -- 172.21.15.82:0/3083936533 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c002ad40 con 0x7f36cc07be10 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.190+0000 7f36cb7fe640 1 -- 172.21.15.82:0/3083936533 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36c0032900 con 0x7f36cc07be10 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.190+0000 7f36cb7fe640 1 -- 172.21.15.82:0/3083936533 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c0030860 con 0x7f36cc07be10 2024-06-06T07:36:32.962 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.191+0000 7f36d31a4640 1 -- 172.21.15.82:0/3083936533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 msgr2=0x7f36cc07a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.191+0000 7f36d31a4640 1 --2- 172.21.15.82:0/3083936533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc07a270 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009820 tx=0x7f36c002a270 comp rx=0 tx=0).stop 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.191+0000 7f36d31a4640 1 -- 172.21.15.82:0/3083936533 shutdown_connections 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.191+0000 7f36d31a4640 1 --2- 172.21.15.82:0/3083936533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc07a270 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.191+0000 7f36d31a4640 1 -- 172.21.15.82:0/3083936533 >> 172.21.15.82:0/3083936533 conn(0x7f36cc1011c0 msgr2=0x7f36cc103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.191+0000 7f36d31a4640 1 -- 172.21.15.82:0/3083936533 shutdown_connections 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.191+0000 7f36d31a4640 1 -- 172.21.15.82:0/3083936533 wait complete. 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d31a4640 1 Processor -- start 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d31a4640 1 -- start start 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d31a4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d31a4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc19bfc0 con 0x7f36cc07be10 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d0f40640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d0f40640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc19ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56928/0 (socket says 172.21.15.82:56928) 2024-06-06T07:36:32.963 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d0f40640 1 -- 172.21.15.82:0/767213168 learned_addr learned my addr 172.21.15.82:0/767213168 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d0f40640 1 -- 172.21.15.82:0/767213168 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c00094d0 con 0x7f36cc07be10 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d0f40640 1 --2- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc19ba80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f36c00097f0 tx=0x7f36c0030e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36c9ffb640 1 -- 172.21.15.82:0/767213168 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c0032c80 con 0x7f36cc07be10 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36cc19c1c0 con 0x7f36cc07be10 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36c9ffb640 1 -- 172.21.15.82:0/767213168 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36c0032de0 con 0x7f36cc07be10 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.192+0000 7f36c9ffb640 1 -- 172.21.15.82:0/767213168 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c003a790 con 0x7f36cc07be10 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.193+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36cc19c660 con 0x7f36cc07be10 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.193+0000 7f36c9ffb640 1 -- 172.21.15.82:0/767213168 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f36c003a9d0 con 0x7f36cc07be10 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.194+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36cc10c080 con 0x7f36cc07be10 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.194+0000 7f36c9ffb640 1 --2- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f36bc037950 0x7f36bc039e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.194+0000 7f36c9ffb640 1 -- 172.21.15.82:0/767213168 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f36c006ac10 con 0x7f36cc07be10 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.194+0000 7f36cbfff640 1 --2- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f36bc037950 0x7f36bc039e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:32.964 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.194+0000 7f36cbfff640 1 --2- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f36bc037950 0x7f36bc039e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f36b80098c0 tx=0x7f36b8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.198+0000 7f36c9ffb640 1 -- 172.21.15.82:0/767213168 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f36c0043310 con 0x7f36cc07be10 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.373+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f36cc10c2c0 con 0x7f36cc07be10 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.373+0000 7f36c9ffb640 1 -- 172.21.15.82:0/767213168 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f36c0037ed0 con 0x7f36cc07be10 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f36bc037950 msgr2=0x7f36bc039e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 --2- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f36bc037950 0x7f36bc039e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f36b80098c0 tx=0x7f36b8006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 msgr2=0x7f36cc19ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 --2- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc19ba80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f36c00097f0 tx=0x7f36c0030e20 comp rx=0 tx=0).stop 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 shutdown_connections 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 --2- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f36bc037950 0x7f36bc039e10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 --2- 172.21.15.82:0/767213168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36cc07be10 0x7f36cc19ba80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 >> 172.21.15.82:0/767213168 conn(0x7f36cc1011c0 msgr2=0x7f36cc101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 shutdown_connections 2024-06-06T07:36:32.965 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:32.375+0000 7f36d31a4640 1 -- 172.21.15.82:0/767213168 wait complete. 2024-06-06T07:36:34.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.385+0000 7fa7f699e640 1 Processor -- start 2024-06-06T07:36:34.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.386+0000 7fa7f699e640 1 -- start start 2024-06-06T07:36:34.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.386+0000 7fa7f699e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f0108a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:34.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.386+0000 7fa7f699e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f0079e40 con 0x7fa7f0106680 2024-06-06T07:36:34.226 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.386+0000 7fa7effff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f0108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.386+0000 7fa7effff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f0108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56934/0 (socket says 172.21.15.82:56934) 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.386+0000 7fa7effff640 1 -- 172.21.15.82:0/464410060 learned_addr learned my addr 172.21.15.82:0/464410060 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.386+0000 7fa7effff640 1 -- 172.21.15.82:0/464410060 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f0108fb0 con 0x7fa7f0106680 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.387+0000 7fa7effff640 1 --2- 172.21.15.82:0/464410060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f0108a70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009820 tx=0x7fa7e002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf564bd877d6bca server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.387+0000 7fa7eeffd640 1 -- 172.21.15.82:0/464410060 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7e002ad40 con 0x7fa7f0106680 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.387+0000 7fa7eeffd640 1 -- 172.21.15.82:0/464410060 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7e0032900 con 0x7fa7f0106680 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.387+0000 7fa7eeffd640 1 -- 172.21.15.82:0/464410060 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7e0030860 con 0x7fa7f0106680 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.388+0000 7fa7f699e640 1 -- 172.21.15.82:0/464410060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 msgr2=0x7fa7f0108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.388+0000 7fa7f699e640 1 --2- 172.21.15.82:0/464410060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f0108a70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009820 tx=0x7fa7e002a270 comp rx=0 tx=0).stop 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.388+0000 7fa7f699e640 1 -- 172.21.15.82:0/464410060 shutdown_connections 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.388+0000 7fa7f699e640 1 --2- 172.21.15.82:0/464410060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f0108a70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.388+0000 7fa7f699e640 1 -- 172.21.15.82:0/464410060 >> 172.21.15.82:0/464410060 conn(0x7fa7f01000d0 msgr2=0x7fa7f01024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:34.227 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.388+0000 7fa7f699e640 1 -- 172.21.15.82:0/464410060 shutdown_connections 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.388+0000 7fa7f699e640 1 -- 172.21.15.82:0/464410060 wait complete. 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.388+0000 7fa7f699e640 1 Processor -- start 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.388+0000 7fa7f699e640 1 -- start start 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.389+0000 7fa7f699e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f01998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.389+0000 7fa7f699e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f0199de0 con 0x7fa7f0106680 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.389+0000 7fa7effff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f01998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.389+0000 7fa7effff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f01998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56942/0 (socket says 172.21.15.82:56942) 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.389+0000 7fa7effff640 1 -- 172.21.15.82:0/416709597 learned_addr learned my addr 172.21.15.82:0/416709597 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.389+0000 7fa7effff640 1 -- 172.21.15.82:0/416709597 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00094d0 con 0x7fa7f0106680 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.389+0000 7fa7effff640 1 --2- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f01998a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00097f0 tx=0x7fa7e0030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.390+0000 7fa7ed7fa640 1 -- 172.21.15.82:0/416709597 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7e0040070 con 0x7fa7f0106680 2024-06-06T07:36:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.390+0000 7fa7ed7fa640 1 -- 172.21.15.82:0/416709597 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7e0032dd0 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.390+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f0199fe0 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.390+0000 7fa7ed7fa640 1 -- 172.21.15.82:0/416709597 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7e003a800 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.390+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f019a480 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.390+0000 7fa7ed7fa640 1 -- 172.21.15.82:0/416709597 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fa7e003a960 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.391+0000 7fa7ed7fa640 1 --2- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fa7d80379a0 0x7fa7d8039e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.391+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f0109e10 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.391+0000 7fa7ed7fa640 1 -- 172.21.15.82:0/416709597 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa7e006a7d0 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.391+0000 7fa7ef7fe640 1 --2- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fa7d80379a0 0x7fa7d8039e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.391+0000 7fa7ef7fe640 1 --2- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fa7d80379a0 0x7fa7d8039e60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40098c0 tx=0x7fa7e4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.395+0000 7fa7ed7fa640 1 -- 172.21.15.82:0/416709597 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa7e00394f0 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.658+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fa7f010a050 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.661+0000 7fa7ed7fa640 1 -- 172.21.15.82:0/416709597 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7fa7e00540c0 con 0x7fa7f0106680 2024-06-06T07:36:34.229 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fa7d80379a0 msgr2=0x7fa7d8039e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 --2- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fa7d80379a0 0x7fa7d8039e60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40098c0 tx=0x7fa7e4006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 msgr2=0x7fa7f01998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 --2- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f01998a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00097f0 tx=0x7fa7e0030e90 comp rx=0 tx=0).stop 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 shutdown_connections 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 --2- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fa7d80379a0 0x7fa7d8039e60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 --2- 172.21.15.82:0/416709597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7f0106680 0x7fa7f01998a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 >> 172.21.15.82:0/416709597 conn(0x7fa7f01000d0 msgr2=0x7fa7f0100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 shutdown_connections 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:/usr/bin/ceph: stderr 2024-06-06T07:36:33.663+0000 7fa7f699e640 1 -- 172.21.15.82:0/416709597 wait complete. 2024-06-06T07:36:34.230 INFO:teuthology.orchestra.run.smithi082.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout:Or, if you are only running a single cluster on this host: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: ceph telemetry on 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout:For more information see: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-06-06T07:36:34.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:34.232 INFO:teuthology.orchestra.run.smithi082.stdout:Bootstrap complete. 2024-06-06T07:36:34.257 INFO:tasks.cephadm:Fetching config... 2024-06-06T07:36:34.258 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:36:34.258 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-06T07:36:34.274 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-06T07:36:34.274 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:36:34.274 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-06T07:36:34.330 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-06T07:36:34.330 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:36:34.330 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/keyring of=/dev/stdout 2024-06-06T07:36:34.395 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-06T07:36:34.395 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:36:34.395 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-06T07:36:34.450 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-06T07:36:34.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8eK7O538orpl7l6DfGO3buZn8qysEtd8J1eG3M4MyBrn3DS7XQH6JfixkBIyFpJrei8qD+DhfFk02A2C7X1pCCHJWx32s/eCPH4r6e3rZrjNGhuoHpM4H6LrQ9TS3SUV28xSfy2+xYZhquiqLrOphMv7QT0Fca+1I10fSqQcffflJ8Govn5dbIoZ51z1AtLAlGF0KqCV6io1uk1+CHo5Hua1jAdxqI+f4k0LSCNR1es2rkP+XsUXJ66UZkdzpRe0d7wC7A7rX0xKA7ncZjmdtyJVYqZZAWHGhf+zBCyq9gnfv71FCzCt8ZYnkpCpoR3zO53e1EMP+M/DCZTXVbpkpg/EAhng++M45lH2s8yP8MgR1BFk/sgH27S1xN1jBRfnjUhkW2omtNJW7Xjrh7JgsW6hiHLY4Gy5M7u3qwURk8VzdVPq5wE/ve1WKDiPMpuMHuw1RIQaY4JE65TiFFI+gnV3T91s8r6f66H8saR1mXfDGbxoHJgqmEMX8SGCUkr0= ceph-355cdb60-23d7-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-06T07:36:34.526 INFO:teuthology.orchestra.run.smithi082.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8eK7O538orpl7l6DfGO3buZn8qysEtd8J1eG3M4MyBrn3DS7XQH6JfixkBIyFpJrei8qD+DhfFk02A2C7X1pCCHJWx32s/eCPH4r6e3rZrjNGhuoHpM4H6LrQ9TS3SUV28xSfy2+xYZhquiqLrOphMv7QT0Fca+1I10fSqQcffflJ8Govn5dbIoZ51z1AtLAlGF0KqCV6io1uk1+CHo5Hua1jAdxqI+f4k0LSCNR1es2rkP+XsUXJ66UZkdzpRe0d7wC7A7rX0xKA7ncZjmdtyJVYqZZAWHGhf+zBCyq9gnfv71FCzCt8ZYnkpCpoR3zO53e1EMP+M/DCZTXVbpkpg/EAhng++M45lH2s8yP8MgR1BFk/sgH27S1xN1jBRfnjUhkW2omtNJW7Xjrh7JgsW6hiHLY4Gy5M7u3qwURk8VzdVPq5wE/ve1WKDiPMpuMHuw1RIQaY4JE65TiFFI+gnV3T91s8r6f66H8saR1mXfDGbxoHJgqmEMX8SGCUkr0= ceph-355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:36:34.538 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8eK7O538orpl7l6DfGO3buZn8qysEtd8J1eG3M4MyBrn3DS7XQH6JfixkBIyFpJrei8qD+DhfFk02A2C7X1pCCHJWx32s/eCPH4r6e3rZrjNGhuoHpM4H6LrQ9TS3SUV28xSfy2+xYZhquiqLrOphMv7QT0Fca+1I10fSqQcffflJ8Govn5dbIoZ51z1AtLAlGF0KqCV6io1uk1+CHo5Hua1jAdxqI+f4k0LSCNR1es2rkP+XsUXJ66UZkdzpRe0d7wC7A7rX0xKA7ncZjmdtyJVYqZZAWHGhf+zBCyq9gnfv71FCzCt8ZYnkpCpoR3zO53e1EMP+M/DCZTXVbpkpg/EAhng++M45lH2s8yP8MgR1BFk/sgH27S1xN1jBRfnjUhkW2omtNJW7Xjrh7JgsW6hiHLY4Gy5M7u3qwURk8VzdVPq5wE/ve1WKDiPMpuMHuw1RIQaY4JE65TiFFI+gnV3T91s8r6f66H8saR1mXfDGbxoHJgqmEMX8SGCUkr0= ceph-355cdb60-23d7-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-06T07:36:34.580 INFO:teuthology.orchestra.run.smithi184.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8eK7O538orpl7l6DfGO3buZn8qysEtd8J1eG3M4MyBrn3DS7XQH6JfixkBIyFpJrei8qD+DhfFk02A2C7X1pCCHJWx32s/eCPH4r6e3rZrjNGhuoHpM4H6LrQ9TS3SUV28xSfy2+xYZhquiqLrOphMv7QT0Fca+1I10fSqQcffflJ8Govn5dbIoZ51z1AtLAlGF0KqCV6io1uk1+CHo5Hua1jAdxqI+f4k0LSCNR1es2rkP+XsUXJ66UZkdzpRe0d7wC7A7rX0xKA7ncZjmdtyJVYqZZAWHGhf+zBCyq9gnfv71FCzCt8ZYnkpCpoR3zO53e1EMP+M/DCZTXVbpkpg/EAhng++M45lH2s8yP8MgR1BFk/sgH27S1xN1jBRfnjUhkW2omtNJW7Xjrh7JgsW6hiHLY4Gy5M7u3qwURk8VzdVPq5wE/ve1WKDiPMpuMHuw1RIQaY4JE65TiFFI+gnV3T91s8r6f66H8saR1mXfDGbxoHJgqmEMX8SGCUkr0= ceph-355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:36:34.593 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-06T07:36:34.746 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:34 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/416709597' entity='client.admin' 2024-06-06T07:36:34.815 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:36:36.525 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.522+0000 7f6fe6de6640 1 -- 172.21.15.82:0/926695341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 msgr2=0x7f6fe0102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:36.525 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.522+0000 7f6fe6de6640 1 --2- 172.21.15.82:0/926695341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 0x7f6fe0102d60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009880 tx=0x7f6fd402a5d0 comp rx=0 tx=0).stop 2024-06-06T07:36:36.525 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.522+0000 7f6fe6de6640 1 -- 172.21.15.82:0/926695341 shutdown_connections 2024-06-06T07:36:36.525 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.522+0000 7f6fe6de6640 1 --2- 172.21.15.82:0/926695341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 0x7f6fe0102d60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.522+0000 7f6fe6de6640 1 -- 172.21.15.82:0/926695341 >> 172.21.15.82:0/926695341 conn(0x7f6fe00fa3a0 msgr2=0x7f6fe00fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.522+0000 7f6fe6de6640 1 -- 172.21.15.82:0/926695341 shutdown_connections 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.522+0000 7f6fe6de6640 1 -- 172.21.15.82:0/926695341 wait complete. 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.523+0000 7f6fe6de6640 1 Processor -- start 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.523+0000 7f6fe6de6640 1 -- start start 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.523+0000 7f6fe6de6640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 0x7f6fe0195400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.523+0000 7f6fe6de6640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe0195940 con 0x7f6fe0100970 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.523+0000 7f6fe4b82640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 0x7f6fe0195400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.523+0000 7f6fe4b82640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 0x7f6fe0195400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:49280/0 (socket says 172.21.15.82:49280) 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.523+0000 7f6fe4b82640 1 -- 172.21.15.82:0/336914349 learned_addr learned my addr 172.21.15.82:0/336914349 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.523+0000 7f6fe4b82640 1 -- 172.21.15.82:0/336914349 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd4009560 con 0x7f6fe0100970 2024-06-06T07:36:36.526 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.523+0000 7f6fe4b82640 1 --2- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 0x7f6fe0195400 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40099a0 tx=0x7f6fd40056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:36.527 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.524+0000 7f6fddffb640 1 -- 172.21.15.82:0/336914349 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd4004aa0 con 0x7f6fe0100970 2024-06-06T07:36:36.527 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.524+0000 7f6fddffb640 1 -- 172.21.15.82:0/336914349 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fd4004c00 con 0x7f6fe0100970 2024-06-06T07:36:36.527 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.524+0000 7f6fddffb640 1 -- 172.21.15.82:0/336914349 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd40319d0 con 0x7f6fe0100970 2024-06-06T07:36:36.527 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.524+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe0195b40 con 0x7f6fe0100970 2024-06-06T07:36:36.527 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.524+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe0195fe0 con 0x7f6fe0100970 2024-06-06T07:36:36.528 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.525+0000 7f6fddffb640 1 -- 172.21.15.82:0/336914349 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f6fd4003cb0 con 0x7f6fe0100970 2024-06-06T07:36:36.528 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.525+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe0061ae0 con 0x7f6fe0100970 2024-06-06T07:36:36.528 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.525+0000 7f6fddffb640 1 --2- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6fd00378e0 0x7f6fd0039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:36.528 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.525+0000 7f6fddffb640 1 -- 172.21.15.82:0/336914349 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6fd406b2d0 con 0x7f6fe0100970 2024-06-06T07:36:36.528 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.525+0000 7f6fdffff640 1 --2- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6fd00378e0 0x7f6fd0039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:36.528 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.525+0000 7f6fdffff640 1 --2- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6fd00378e0 0x7f6fd0039da0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0098c0 tx=0x7f6fcc006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.530+0000 7f6fddffb640 1 -- 172.21.15.82:0/336914349 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6fd40035f0 con 0x7f6fe0100970 2024-06-06T07:36:36.706 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.705+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f6fe0106030 con 0x7f6fe0100970 2024-06-06T07:36:36.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.708+0000 7f6fddffb640 1 -- 172.21.15.82:0/336914349 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6fd4035020 con 0x7f6fe0100970 2024-06-06T07:36:36.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6fd00378e0 msgr2=0x7f6fd0039da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:36.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 --2- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6fd00378e0 0x7f6fd0039da0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0098c0 tx=0x7f6fcc006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:36.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 msgr2=0x7f6fe0195400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:36.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 --2- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 0x7f6fe0195400 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40099a0 tx=0x7f6fd40056e0 comp rx=0 tx=0).stop 2024-06-06T07:36:36.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 shutdown_connections 2024-06-06T07:36:36.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 --2- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6fd00378e0 0x7f6fd0039da0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:36.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 --2- 172.21.15.82:0/336914349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fe0100970 0x7f6fe0195400 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:36.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 >> 172.21.15.82:0/336914349 conn(0x7f6fe00fa3a0 msgr2=0x7f6fe00fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:36.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 shutdown_connections 2024-06-06T07:36:36.713 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:36.710+0000 7f6fe6de6640 1 -- 172.21.15.82:0/336914349 wait complete. 2024-06-06T07:36:37.569 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-06T07:36:37.570 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-06T07:36:37.823 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:37 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/336914349' entity='client.admin' 2024-06-06T07:36:37.823 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:37 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:37.823 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:37 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:37.823 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:37 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:36:37.823 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:37 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:37.823 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:37 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi082", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-06T07:36:37.823 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:37 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi082", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-06-06T07:36:37.824 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:37 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:36:37.824 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:37 smithi082 ceph-mon[29354]: Deploying daemon ceph-exporter.smithi082 on smithi082 2024-06-06T07:36:38.064 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:36:40.188 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.181+0000 7f2941329640 1 -- 172.21.15.82:0/3871811225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 msgr2=0x7f293c0ff1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:40.188 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.181+0000 7f2941329640 1 --2- 172.21.15.82:0/3871811225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 0x7f293c0ff1f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f292c009880 tx=0x7f292c02a5d0 comp rx=0 tx=0).stop 2024-06-06T07:36:40.188 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.181+0000 7f2941329640 1 -- 172.21.15.82:0/3871811225 shutdown_connections 2024-06-06T07:36:40.188 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.181+0000 7f2941329640 1 --2- 172.21.15.82:0/3871811225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 0x7f293c0ff1f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:40.188 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.181+0000 7f2941329640 1 -- 172.21.15.82:0/3871811225 >> 172.21.15.82:0/3871811225 conn(0x7f293c0fa370 msgr2=0x7f293c0fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:40.188 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.181+0000 7f2941329640 1 -- 172.21.15.82:0/3871811225 shutdown_connections 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.181+0000 7f2941329640 1 -- 172.21.15.82:0/3871811225 wait complete. 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.181+0000 7f2941329640 1 Processor -- start 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f2941329640 1 -- start start 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f2941329640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 0x7f293c195400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f2941329640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c195940 con 0x7f293c0fedd0 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f293affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 0x7f293c195400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f293affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 0x7f293c195400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:49292/0 (socket says 172.21.15.82:49292) 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f293affd640 1 -- 172.21.15.82:0/3459251177 learned_addr learned my addr 172.21.15.82:0/3459251177 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f293affd640 1 -- 172.21.15.82:0/3459251177 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c009560 con 0x7f293c0fedd0 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f293affd640 1 --2- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 0x7f293c195400 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f292c005610 tx=0x7f292c005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f2923fff640 1 -- 172.21.15.82:0/3459251177 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f292c004b20 con 0x7f293c0fedd0 2024-06-06T07:36:40.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c195b40 con 0x7f293c0fedd0 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f2923fff640 1 -- 172.21.15.82:0/3459251177 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f292c004c80 con 0x7f293c0fedd0 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f2923fff640 1 -- 172.21.15.82:0/3459251177 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f292c031a90 con 0x7f293c0fedd0 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.182+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c195fe0 con 0x7f293c0fedd0 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.183+0000 7f2923fff640 1 -- 172.21.15.82:0/3459251177 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f292c031d10 con 0x7f293c0fedd0 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.184+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c061ae0 con 0x7f293c0fedd0 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.184+0000 7f2923fff640 1 --2- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2924037890 0x7f2924039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.184+0000 7f2923fff640 1 -- 172.21.15.82:0/3459251177 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f292c06b370 con 0x7f293c0fedd0 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.184+0000 7f293a7fc640 1 --2- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2924037890 0x7f2924039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.184+0000 7f293a7fc640 1 --2- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2924037890 0x7f2924039d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f29300098c0 tx=0x7f2930006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:40.190 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.188+0000 7f2923fff640 1 -- 172.21.15.82:0/3459251177 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f292c04d9a0 con 0x7f293c0fedd0 2024-06-06T07:36:40.371 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.369+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 --> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f293c068960 con 0x7f2924037890 2024-06-06T07:36:40.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.372+0000 7f2923fff640 1 -- 172.21.15.82:0/3459251177 <== mgr.14160 v2:172.21.15.82:6800/493557010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f293c068960 con 0x7f2924037890 2024-06-06T07:36:40.376 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2924037890 msgr2=0x7f2924039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:40.376 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 --2- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2924037890 0x7f2924039d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f29300098c0 tx=0x7f2930006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:40.376 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 msgr2=0x7f293c195400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:40.376 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 --2- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 0x7f293c195400 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f292c005610 tx=0x7f292c005640 comp rx=0 tx=0).stop 2024-06-06T07:36:40.376 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 shutdown_connections 2024-06-06T07:36:40.376 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 --2- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2924037890 0x7f2924039d50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:40.376 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 --2- 172.21.15.82:0/3459251177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f293c0fedd0 0x7f293c195400 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:40.377 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 >> 172.21.15.82:0/3459251177 conn(0x7f293c0fa370 msgr2=0x7f293c068250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:40.377 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 shutdown_connections 2024-06-06T07:36:40.377 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:40.374+0000 7f2941329640 1 -- 172.21.15.82:0/3459251177 wait complete. 2024-06-06T07:36:41.365 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi184 2024-06-06T07:36:41.365 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:36:41.365 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.conf 2024-06-06T07:36:41.385 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:36:41.385 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-06T07:36:41.443 INFO:tasks.cephadm:Adding host smithi184 to orchestrator... 2024-06-06T07:36:41.443 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch host add smithi184 2024-06-06T07:36:41.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:41 smithi082 ceph-mon[29354]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:41.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:41 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:41.879 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:36:43.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.509+0000 7f5833250640 1 -- 172.21.15.82:0/613393093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 msgr2=0x7f582c102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:43.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.509+0000 7f5833250640 1 --2- 172.21.15.82:0/613393093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 0x7f582c102db0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5820009880 tx=0x7f582002a5d0 comp rx=0 tx=0).stop 2024-06-06T07:36:43.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.509+0000 7f5833250640 1 -- 172.21.15.82:0/613393093 shutdown_connections 2024-06-06T07:36:43.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.509+0000 7f5833250640 1 --2- 172.21.15.82:0/613393093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 0x7f582c102db0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:43.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.509+0000 7f5833250640 1 -- 172.21.15.82:0/613393093 >> 172.21.15.82:0/613393093 conn(0x7f582c0fa370 msgr2=0x7f582c0fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:43.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.509+0000 7f5833250640 1 -- 172.21.15.82:0/613393093 shutdown_connections 2024-06-06T07:36:43.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.509+0000 7f5833250640 1 -- 172.21.15.82:0/613393093 wait complete. 2024-06-06T07:36:43.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.510+0000 7f5833250640 1 Processor -- start 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.510+0000 7f5833250640 1 -- start start 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.510+0000 7f5833250640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 0x7f582c1001f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.510+0000 7f5833250640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c100730 con 0x7f582c1009c0 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.510+0000 7f5830fec640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 0x7f582c1001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.510+0000 7f5830fec640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 0x7f582c1001f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:49306/0 (socket says 172.21.15.82:49306) 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.510+0000 7f5830fec640 1 -- 172.21.15.82:0/1864069669 learned_addr learned my addr 172.21.15.82:0/1864069669 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.510+0000 7f5830fec640 1 -- 172.21.15.82:0/1864069669 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5820009560 con 0x7f582c1009c0 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.510+0000 7f5830fec640 1 --2- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 0x7f582c1001f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5820005610 tx=0x7f5820005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.511+0000 7f5829ffb640 1 -- 172.21.15.82:0/1864069669 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5820004be0 con 0x7f582c1009c0 2024-06-06T07:36:43.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.511+0000 7f5829ffb640 1 -- 172.21.15.82:0/1864069669 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5820004d40 con 0x7f582c1009c0 2024-06-06T07:36:43.514 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.511+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c0fe970 con 0x7f582c1009c0 2024-06-06T07:36:43.514 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.511+0000 7f5829ffb640 1 -- 172.21.15.82:0/1864069669 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5820031b50 con 0x7f582c1009c0 2024-06-06T07:36:43.514 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.511+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c0fee10 con 0x7f582c1009c0 2024-06-06T07:36:43.514 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.512+0000 7f5829ffb640 1 -- 172.21.15.82:0/1864069669 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f5820031cb0 con 0x7f582c1009c0 2024-06-06T07:36:43.515 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.512+0000 7f5829ffb640 1 --2- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f581c0378e0 0x7f581c039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:43.515 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.512+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c106690 con 0x7f582c1009c0 2024-06-06T07:36:43.515 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.512+0000 7f5829ffb640 1 -- 172.21.15.82:0/1864069669 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f582006b350 con 0x7f582c1009c0 2024-06-06T07:36:43.515 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.512+0000 7f582bfff640 1 --2- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f581c0378e0 0x7f581c039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:43.515 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.512+0000 7f582bfff640 1 --2- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f581c0378e0 0x7f581c039da0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f58180098c0 tx=0x7f5818006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:43.517 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.516+0000 7f5829ffb640 1 -- 172.21.15.82:0/1864069669 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5820040080 con 0x7f582c1009c0 2024-06-06T07:36:43.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:43.713+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 --> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}) v1 -- 0x7f582c18e7b0 con 0x7f581c0378e0 2024-06-06T07:36:45.420 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:45 smithi082 ceph-mon[29354]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:45.421 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:45 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:45.421 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:45 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:45.421 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:45 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:45.421 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:45 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:45.421 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:45 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi082", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T07:36:45.421 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:45 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi082", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-06T07:36:45.421 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:45 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:36:45.421 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:45 smithi082 ceph-mon[29354]: Deploying daemon crash.smithi082 on smithi082 2024-06-06T07:36:46.403 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.400+0000 7f5829ffb640 1 -- 172.21.15.82:0/1864069669 <== mgr.14160 v2:172.21.15.82:6800/493557010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f582c18e7b0 con 0x7f581c0378e0 2024-06-06T07:36:46.403 INFO:teuthology.orchestra.run.smithi082.stdout:Added host 'smithi184' with addr '172.21.15.184' 2024-06-06T07:36:46.404 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f581c0378e0 msgr2=0x7f581c039da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:46.405 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 --2- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f581c0378e0 0x7f581c039da0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f58180098c0 tx=0x7f5818006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:46.405 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 msgr2=0x7f582c1001f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:46.405 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 --2- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 0x7f582c1001f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5820005610 tx=0x7f5820005640 comp rx=0 tx=0).stop 2024-06-06T07:36:46.405 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 shutdown_connections 2024-06-06T07:36:46.405 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 --2- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f581c0378e0 0x7f581c039da0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:46.405 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 --2- 172.21.15.82:0/1864069669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f582c1009c0 0x7f582c1001f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:46.405 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 >> 172.21.15.82:0/1864069669 conn(0x7f582c0fa370 msgr2=0x7f582c0fae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:46.405 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 shutdown_connections 2024-06-06T07:36:46.406 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:46.404+0000 7f5833250640 1 -- 172.21.15.82:0/1864069669 wait complete. 2024-06-06T07:36:46.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:46 smithi082 ceph-mon[29354]: Deploying cephadm binary to smithi184 2024-06-06T07:36:46.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:46 smithi082 ceph-mon[29354]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:46.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:46 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:47.094 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-06T07:36:47.328 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:36:47.645 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:47.645 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: Added host smithi184 2024-06-06T07:36:47.645 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:47.645 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:47.645 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:47.645 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:47.646 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: Deploying daemon node-exporter.smithi082 on smithi082 2024-06-06T07:36:47.646 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:47.646 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:47.646 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:47 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:48.752 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:48 smithi082 ceph-mon[29354]: from='client.14189 -' entity='client.crash.smithi082' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:48.752 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:48 smithi082 ceph-mon[29354]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:48.752 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:48 smithi082 ceph-mon[29354]: from='client.14191 -' entity='client.crash.smithi082' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:48.753 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:48 smithi082 ceph-mon[29354]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:48.753 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:48 smithi082 ceph-mon[29354]: from='client.14193 -' entity='client.crash.smithi082' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:48.753 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:48 smithi082 ceph-mon[29354]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:48.753 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:48 smithi082 ceph-mon[29354]: Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-06T07:36:48.954 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.954+0000 7f71a8b23640 1 -- 172.21.15.82:0/1447538587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 msgr2=0x7f71a40ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:48.954 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.954+0000 7f71a8b23640 1 --2- 172.21.15.82:0/1447538587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 0x7f71a40ff200 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7194009c20 tx=0x7f719402aa90 comp rx=0 tx=0).stop 2024-06-06T07:36:48.955 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.954+0000 7f71a8b23640 1 -- 172.21.15.82:0/1447538587 shutdown_connections 2024-06-06T07:36:48.955 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.954+0000 7f71a8b23640 1 --2- 172.21.15.82:0/1447538587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 0x7f71a40ff200 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:48.955 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.954+0000 7f71a8b23640 1 -- 172.21.15.82:0/1447538587 >> 172.21.15.82:0/1447538587 conn(0x7f71a40fa380 msgr2=0x7f71a40fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:48.955 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.954+0000 7f71a8b23640 1 -- 172.21.15.82:0/1447538587 shutdown_connections 2024-06-06T07:36:48.955 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.954+0000 7f71a8b23640 1 -- 172.21.15.82:0/1447538587 wait complete. 2024-06-06T07:36:48.956 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.954+0000 7f71a8b23640 1 Processor -- start 2024-06-06T07:36:48.956 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a8b23640 1 -- start start 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a8b23640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 0x7f71a40fc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a8b23640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40fc6e0 con 0x7f71a40fede0 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 0x7f71a40fc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 0x7f71a40fc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52782/0 (socket says 172.21.15.82:52782) 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a259c640 1 -- 172.21.15.82:0/3989322370 learned_addr learned my addr 172.21.15.82:0/3989322370 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a259c640 1 -- 172.21.15.82:0/3989322370 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7194009900 con 0x7f71a40fede0 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a259c640 1 --2- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 0x7f71a40fc1a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7194005cc0 tx=0x7f719402e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71937fe640 1 -- 172.21.15.82:0/3989322370 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f719403e070 con 0x7f71a40fede0 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71937fe640 1 -- 172.21.15.82:0/3989322370 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f719402e990 con 0x7f71a40fede0 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a40fa970 con 0x7f71a40fede0 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71937fe640 1 -- 172.21.15.82:0/3989322370 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7194038450 con 0x7f71a40fede0 2024-06-06T07:36:48.957 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.955+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a40fae10 con 0x7f71a40fede0 2024-06-06T07:36:48.958 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.956+0000 7f71937fe640 1 -- 172.21.15.82:0/3989322370 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f719400a040 con 0x7f71a40fede0 2024-06-06T07:36:48.958 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.956+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a40fc820 con 0x7f71a40fede0 2024-06-06T07:36:48.958 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.957+0000 7f71937fe640 1 --2- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f718c037890 0x7f718c039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:48.958 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.957+0000 7f71937fe640 1 -- 172.21.15.82:0/3989322370 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f719406b380 con 0x7f71a40fede0 2024-06-06T07:36:48.959 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.957+0000 7f71a1d9b640 1 --2- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f718c037890 0x7f718c039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:48.959 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.957+0000 7f71a1d9b640 1 --2- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f718c037890 0x7f718c039d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f71980098c0 tx=0x7f7198006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:48.962 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:48.961+0000 7f71937fe640 1 -- 172.21.15.82:0/3989322370 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f71a40fc820 con 0x7f71a40fede0 2024-06-06T07:36:49.156 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.155+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 --> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71a40fc820 con 0x7f718c037890 2024-06-06T07:36:49.156 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.156+0000 7f71937fe640 1 -- 172.21.15.82:0/3989322370 <== mgr.14160 v2:172.21.15.82:6800/493557010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f71a40fc820 con 0x7f718c037890 2024-06-06T07:36:49.157 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:36:49.157 INFO:teuthology.orchestra.run.smithi082.stdout:[{"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}, {"addr": "172.21.15.184", "hostname": "smithi184", "labels": [], "status": ""}] 2024-06-06T07:36:49.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f718c037890 msgr2=0x7f718c039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:49.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 --2- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f718c037890 0x7f718c039d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f71980098c0 tx=0x7f7198006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:49.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 msgr2=0x7f71a40fc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:49.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 --2- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 0x7f71a40fc1a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7194005cc0 tx=0x7f719402e5f0 comp rx=0 tx=0).stop 2024-06-06T07:36:49.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 shutdown_connections 2024-06-06T07:36:49.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 --2- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f718c037890 0x7f718c039d50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:49.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 --2- 172.21.15.82:0/3989322370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71a40fede0 0x7f71a40fc1a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:49.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 >> 172.21.15.82:0/3989322370 conn(0x7f71a40fa380 msgr2=0x7f71a4067b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:49.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 shutdown_connections 2024-06-06T07:36:49.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:49.158+0000 7f71a8b23640 1 -- 172.21.15.82:0/3989322370 wait complete. 2024-06-06T07:36:50.227 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-06T07:36:50.464 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-06-06T07:36:50.493 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:50 smithi082 ceph-mon[29354]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T07:36:50.493 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:50 smithi082 ceph-mon[29354]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:50.493 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:50 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:50.717 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:36:52.504 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:52 smithi082 ceph-mon[29354]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:52.504 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:52 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:52.504 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:52 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:52.504 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:52 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:52.504 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:52 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.007+0000 7fac76065640 1 -- 172.21.15.82:0/2765204988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 msgr2=0x7fac700ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.007+0000 7fac76065640 1 --2- 172.21.15.82:0/2765204988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 0x7fac700ff200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fac60009880 tx=0x7fac6002a5d0 comp rx=0 tx=0).stop 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.007+0000 7fac76065640 1 -- 172.21.15.82:0/2765204988 shutdown_connections 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.007+0000 7fac76065640 1 --2- 172.21.15.82:0/2765204988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 0x7fac700ff200 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.007+0000 7fac76065640 1 -- 172.21.15.82:0/2765204988 >> 172.21.15.82:0/2765204988 conn(0x7fac700fa380 msgr2=0x7fac700fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.007+0000 7fac76065640 1 -- 172.21.15.82:0/2765204988 shutdown_connections 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.007+0000 7fac76065640 1 -- 172.21.15.82:0/2765204988 wait complete. 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.007+0000 7fac76065640 1 Processor -- start 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac76065640 1 -- start start 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac76065640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 0x7fac70195400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:53.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac76065640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac70195940 con 0x7fac700fede0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac6f7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 0x7fac70195400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac6f7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 0x7fac70195400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52808/0 (socket says 172.21.15.82:52808) 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac6f7fe640 1 -- 172.21.15.82:0/1248535075 learned_addr learned my addr 172.21.15.82:0/1248535075 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac6f7fe640 1 -- 172.21.15.82:0/1248535075 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac60009560 con 0x7fac700fede0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac6f7fe640 1 --2- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 0x7fac70195400 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fac60005610 tx=0x7fac60005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac6cff9640 1 -- 172.21.15.82:0/1248535075 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac60004b20 con 0x7fac700fede0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac70195b40 con 0x7fac700fede0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac6cff9640 1 -- 172.21.15.82:0/1248535075 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac60004c80 con 0x7fac700fede0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.008+0000 7fac6cff9640 1 -- 172.21.15.82:0/1248535075 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac60031a90 con 0x7fac700fede0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.009+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac70195fe0 con 0x7fac700fede0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.009+0000 7fac6cff9640 1 -- 172.21.15.82:0/1248535075 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fac60031d10 con 0x7fac700fede0 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.009+0000 7fac6cff9640 1 --2- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fac5c037890 0x7fac5c039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:53.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.009+0000 7fac6cff9640 1 -- 172.21.15.82:0/1248535075 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fac6006b630 con 0x7fac700fede0 2024-06-06T07:36:53.018 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.009+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac70061ae0 con 0x7fac700fede0 2024-06-06T07:36:53.018 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.010+0000 7fac6effd640 1 --2- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fac5c037890 0x7fac5c039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:53.018 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.010+0000 7fac6effd640 1 --2- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fac5c037890 0x7fac5c039d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fac580098c0 tx=0x7fac58006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:53.018 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.014+0000 7fac6cff9640 1 -- 172.21.15.82:0/1248535075 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fac60035070 con 0x7fac700fede0 2024-06-06T07:36:53.189 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.189+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fac70104ca0 con 0x7fac700fede0 2024-06-06T07:36:53.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.415+0000 7fac6cff9640 1 -- 172.21.15.82:0/1248535075 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fac60043360 con 0x7fac700fede0 2024-06-06T07:36:53.416 INFO:teuthology.orchestra.run.smithi082.stderr:adjusted tunables profile to default 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fac5c037890 msgr2=0x7fac5c039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 --2- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fac5c037890 0x7fac5c039d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fac580098c0 tx=0x7fac58006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 msgr2=0x7fac70195400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 --2- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 0x7fac70195400 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fac60005610 tx=0x7fac60005640 comp rx=0 tx=0).stop 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 shutdown_connections 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 --2- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fac5c037890 0x7fac5c039d50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 --2- 172.21.15.82:0/1248535075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac700fede0 0x7fac70195400 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 >> 172.21.15.82:0/1248535075 conn(0x7fac700fa380 msgr2=0x7fac700fb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 shutdown_connections 2024-06-06T07:36:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:36:53.417+0000 7fac76065640 1 -- 172.21.15.82:0/1248535075 wait complete. 2024-06-06T07:36:53.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:53 smithi082 ceph-mon[29354]: Deploying daemon alertmanager.smithi082 on smithi082 2024-06-06T07:36:53.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:53 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1248535075' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-06T07:36:54.570 INFO:tasks.cephadm:Adding mon.smithi082 on smithi082 2024-06-06T07:36:54.571 INFO:tasks.cephadm:Adding mon.smithi184 on smithi184 2024-06-06T07:36:54.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch apply mon '2;smithi082:172.21.15.82=smithi082;smithi184:172.21.15.184=smithi184' 2024-06-06T07:36:54.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:54 smithi082 ceph-mon[29354]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:54.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:54 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1248535075' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-06T07:36:54.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:54 smithi082 ceph-mon[29354]: osdmap e4: 0 total, 0 up, 0 in 2024-06-06T07:36:54.770 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:36:54.817 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:36:56.241 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.240+0000 7fda273b8640 1 -- 172.21.15.184:0/243353361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 msgr2=0x7fda20102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:56.241 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.240+0000 7fda273b8640 1 --2- 172.21.15.184:0/243353361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 0x7fda20102de0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fda1c009880 tx=0x7fda1c02a5d0 comp rx=0 tx=0).stop 2024-06-06T07:36:56.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.240+0000 7fda273b8640 1 -- 172.21.15.184:0/243353361 shutdown_connections 2024-06-06T07:36:56.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.240+0000 7fda273b8640 1 --2- 172.21.15.184:0/243353361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 0x7fda20102de0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:56.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.240+0000 7fda273b8640 1 -- 172.21.15.184:0/243353361 >> 172.21.15.184:0/243353361 conn(0x7fda200fa3a0 msgr2=0x7fda200fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:56.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.240+0000 7fda273b8640 1 -- 172.21.15.184:0/243353361 shutdown_connections 2024-06-06T07:36:56.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.240+0000 7fda273b8640 1 -- 172.21.15.184:0/243353361 wait complete. 2024-06-06T07:36:56.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.241+0000 7fda273b8640 1 Processor -- start 2024-06-06T07:36:56.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.241+0000 7fda273b8640 1 -- start start 2024-06-06T07:36:56.243 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.241+0000 7fda273b8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 0x7fda20195390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:56.243 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.241+0000 7fda273b8640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201958d0 con 0x7fda201009f0 2024-06-06T07:36:56.243 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda25154640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 0x7fda20195390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:56.243 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda25154640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 0x7fda20195390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:59102/0 (socket says 172.21.15.184:59102) 2024-06-06T07:36:56.243 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda25154640 1 -- 172.21.15.184:0/407154343 learned_addr learned my addr 172.21.15.184:0/407154343 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:36:56.243 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda25154640 1 -- 172.21.15.184:0/407154343 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c009560 con 0x7fda201009f0 2024-06-06T07:36:56.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda25154640 1 --2- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 0x7fda20195390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fda1c006010 tx=0x7fda1c005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:56.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda167fc640 1 -- 172.21.15.184:0/407154343 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1c004aa0 con 0x7fda201009f0 2024-06-06T07:36:56.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda20195ad0 con 0x7fda201009f0 2024-06-06T07:36:56.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda167fc640 1 -- 172.21.15.184:0/407154343 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda1c004c00 con 0x7fda201009f0 2024-06-06T07:36:56.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda167fc640 1 -- 172.21.15.184:0/407154343 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1c031a80 con 0x7fda201009f0 2024-06-06T07:36:56.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.242+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda20195f70 con 0x7fda201009f0 2024-06-06T07:36:56.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.243+0000 7fda167fc640 1 -- 172.21.15.184:0/407154343 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fda1c031d40 con 0x7fda201009f0 2024-06-06T07:36:56.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.243+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20061ae0 con 0x7fda201009f0 2024-06-06T07:36:56.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.243+0000 7fda167fc640 1 --2- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fda0c037890 0x7fda0c039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:56.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.243+0000 7fda167fc640 1 -- 172.21.15.184:0/407154343 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fda1c06b4a0 con 0x7fda201009f0 2024-06-06T07:36:56.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.244+0000 7fda24953640 1 --2- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fda0c037890 0x7fda0c039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:56.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.244+0000 7fda24953640 1 --2- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fda0c037890 0x7fda0c039d50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fda100098c0 tx=0x7fda10006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:56.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.248+0000 7fda167fc640 1 -- 172.21.15.184:0/407154343 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fda1c04d9a0 con 0x7fda201009f0 2024-06-06T07:36:56.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.430+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 --> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi082:172.21.15.82=smithi082;smithi184:172.21.15.184=smithi184", "target": ["mon-mgr", ""]}) v1 -- 0x7fda2002c5c0 con 0x7fda0c037890 2024-06-06T07:36:56.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.435+0000 7fda167fc640 1 -- 172.21.15.184:0/407154343 <== mgr.14160 v2:172.21.15.82:6800/493557010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fda2002c5c0 con 0x7fda0c037890 2024-06-06T07:36:56.435 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mon update... 2024-06-06T07:36:56.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fda0c037890 msgr2=0x7fda0c039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:56.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 --2- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fda0c037890 0x7fda0c039d50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fda100098c0 tx=0x7fda10006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:56.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 msgr2=0x7fda20195390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:56.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 --2- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 0x7fda20195390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fda1c006010 tx=0x7fda1c005630 comp rx=0 tx=0).stop 2024-06-06T07:36:56.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 shutdown_connections 2024-06-06T07:36:56.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 --2- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fda0c037890 0x7fda0c039d50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:56.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 --2- 172.21.15.184:0/407154343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda201009f0 0x7fda20195390 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:56.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 >> 172.21.15.184:0/407154343 conn(0x7fda200fa3a0 msgr2=0x7fda200fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:56.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 shutdown_connections 2024-06-06T07:36:56.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:56.437+0000 7fda273b8640 1 -- 172.21.15.184:0/407154343 wait complete. 2024-06-06T07:36:56.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:56 smithi082 ceph-mon[29354]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:56.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:56 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:57.026 DEBUG:teuthology.orchestra.run.smithi184:mon.smithi184> sudo journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi184.service 2024-06-06T07:36:57.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:36:57.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:36:57.254 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:36:57.301 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:36:57.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:57 smithi082 ceph-mon[29354]: from='client.14199 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi082:172.21.15.82=smithi082;smithi184:172.21.15.184=smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:36:57.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:57 smithi082 ceph-mon[29354]: Saving service mon spec with placement smithi082:172.21.15.82=smithi082;smithi184:172.21.15.184=smithi184;count:2 2024-06-06T07:36:57.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:57 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:36:58.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:58 smithi082 ceph-mon[29354]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:36:58.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.749+0000 7fe78f88e640 1 -- 172.21.15.184:0/2987574663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 msgr2=0x7fe7880feb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.749+0000 7fe78f88e640 1 --2- 172.21.15.184:0/2987574663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 0x7fe7880feb80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe784009880 tx=0x7fe78402a5d0 comp rx=0 tx=0).stop 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.749+0000 7fe78f88e640 1 -- 172.21.15.184:0/2987574663 shutdown_connections 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.749+0000 7fe78f88e640 1 --2- 172.21.15.184:0/2987574663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 0x7fe7880feb80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.749+0000 7fe78f88e640 1 -- 172.21.15.184:0/2987574663 >> 172.21.15.184:0/2987574663 conn(0x7fe7880fa380 msgr2=0x7fe7880fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.749+0000 7fe78f88e640 1 -- 172.21.15.184:0/2987574663 shutdown_connections 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.749+0000 7fe78f88e640 1 -- 172.21.15.184:0/2987574663 wait complete. 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.750+0000 7fe78f88e640 1 Processor -- start 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.750+0000 7fe78f88e640 1 -- start start 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.750+0000 7fe78f88e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 0x7fe7881975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.750+0000 7fe78f88e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe788197af0 con 0x7fe7880fe760 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.750+0000 7fe78d62a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 0x7fe7881975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.750+0000 7fe78d62a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 0x7fe7881975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:59120/0 (socket says 172.21.15.184:59120) 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.750+0000 7fe78d62a640 1 -- 172.21.15.184:0/3282633980 learned_addr learned my addr 172.21.15.184:0/3282633980 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.751+0000 7fe78d62a640 1 -- 172.21.15.184:0/3282633980 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe784009560 con 0x7fe7880fe760 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.751+0000 7fe78d62a640 1 --2- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 0x7fe7881975b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe784006010 tx=0x7fe784005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.751+0000 7fe77e7fc640 1 -- 172.21.15.184:0/3282633980 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe784004aa0 con 0x7fe7880fe760 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.751+0000 7fe77e7fc640 1 -- 172.21.15.184:0/3282633980 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe784004c00 con 0x7fe7880fe760 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.751+0000 7fe77e7fc640 1 -- 172.21.15.184:0/3282633980 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe784031a80 con 0x7fe7880fe760 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.751+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe788197cf0 con 0x7fe7880fe760 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.751+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe788198190 con 0x7fe7880fe760 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.752+0000 7fe77e7fc640 1 -- 172.21.15.184:0/3282633980 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fe784031d40 con 0x7fe7880fe760 2024-06-06T07:36:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.752+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe788061ae0 con 0x7fe7880fe760 2024-06-06T07:36:58.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.752+0000 7fe77e7fc640 1 --2- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe774037890 0x7fe774039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:36:58.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.752+0000 7fe77e7fc640 1 -- 172.21.15.184:0/3282633980 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe78406b4a0 con 0x7fe7880fe760 2024-06-06T07:36:58.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.752+0000 7fe78ce29640 1 --2- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe774037890 0x7fe774039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:36:58.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.753+0000 7fe78ce29640 1 --2- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe774037890 0x7fe774039d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe7780098c0 tx=0x7fe778006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:36:58.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.757+0000 7fe77e7fc640 1 -- 172.21.15.184:0/3282633980 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe78404d9a0 con 0x7fe7880fe760 2024-06-06T07:36:58.995 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.995+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe78802c3e0 con 0x7fe7880fe760 2024-06-06T07:36:58.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.996+0000 7fe77e7fc640 1 -- 172.21.15.184:0/3282633980 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fe784035070 con 0x7fe7880fe760 2024-06-06T07:36:58.996 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:36:58.996 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:36:58.997 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:36:58.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe774037890 msgr2=0x7fe774039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:58.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 --2- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe774037890 0x7fe774039d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe7780098c0 tx=0x7fe778006db0 comp rx=0 tx=0).stop 2024-06-06T07:36:58.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 msgr2=0x7fe7881975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:36:58.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 --2- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 0x7fe7881975b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe784006010 tx=0x7fe784005630 comp rx=0 tx=0).stop 2024-06-06T07:36:58.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 shutdown_connections 2024-06-06T07:36:58.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 --2- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fe774037890 0x7fe774039d50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:58.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 --2- 172.21.15.184:0/3282633980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7880fe760 0x7fe7881975b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:36:58.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 >> 172.21.15.184:0/3282633980 conn(0x7fe7880fa380 msgr2=0x7fe7880fae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:36:58.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 shutdown_connections 2024-06-06T07:36:58.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:36:58.998+0000 7fe78f88e640 1 -- 172.21.15.184:0/3282633980 wait complete. 2024-06-06T07:36:59.596 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:36:59 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/3282633980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:00.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:00.561 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:00.624 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:00 smithi082 ceph-mon[29354]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:00.758 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:00.805 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.174+0000 7fdff0f54640 1 -- 172.21.15.184:0/3963461681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 msgr2=0x7fdfec100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.174+0000 7fdff0f54640 1 --2- 172.21.15.184:0/3963461681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 0x7fdfec100b80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009880 tx=0x7fdfdc02a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.174+0000 7fdff0f54640 1 -- 172.21.15.184:0/3963461681 shutdown_connections 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.174+0000 7fdff0f54640 1 --2- 172.21.15.184:0/3963461681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 0x7fdfec100b80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.174+0000 7fdff0f54640 1 -- 172.21.15.184:0/3963461681 >> 172.21.15.184:0/3963461681 conn(0x7fdfec0fa370 msgr2=0x7fdfec0fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.174+0000 7fdff0f54640 1 -- 172.21.15.184:0/3963461681 shutdown_connections 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.174+0000 7fdff0f54640 1 -- 172.21.15.184:0/3963461681 wait complete. 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.175+0000 7fdff0f54640 1 Processor -- start 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.175+0000 7fdff0f54640 1 -- start start 2024-06-06T07:37:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.175+0000 7fdff0f54640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 0x7fdfec19b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:02.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.175+0000 7fdff0f54640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec19bef0 con 0x7fdfec0fe750 2024-06-06T07:37:02.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdfea59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 0x7fdfec19b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:02.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdfea59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 0x7fdfec19b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:59140/0 (socket says 172.21.15.184:59140) 2024-06-06T07:37:02.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdfea59c640 1 -- 172.21.15.184:0/2460142609 learned_addr learned my addr 172.21.15.184:0/2460142609 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:02.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdfea59c640 1 -- 172.21.15.184:0/2460142609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc009560 con 0x7fdfec0fe750 2024-06-06T07:37:02.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdfea59c640 1 --2- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 0x7fdfec19b9b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc006010 tx=0x7fdfdc005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:02.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdfe37fe640 1 -- 172.21.15.184:0/2460142609 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfdc004aa0 con 0x7fdfec0fe750 2024-06-06T07:37:02.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfec19c0f0 con 0x7fdfec0fe750 2024-06-06T07:37:02.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdfe37fe640 1 -- 172.21.15.184:0/2460142609 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfdc004c00 con 0x7fdfec0fe750 2024-06-06T07:37:02.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdfe37fe640 1 -- 172.21.15.184:0/2460142609 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfdc031a80 con 0x7fdfec0fe750 2024-06-06T07:37:02.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.176+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfec19c590 con 0x7fdfec0fe750 2024-06-06T07:37:02.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.177+0000 7fdfe37fe640 1 -- 172.21.15.184:0/2460142609 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fdfdc031d40 con 0x7fdfec0fe750 2024-06-06T07:37:02.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.177+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfec061ae0 con 0x7fdfec0fe750 2024-06-06T07:37:02.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.178+0000 7fdfe37fe640 1 --2- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fdfd8037890 0x7fdfd8039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:02.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.178+0000 7fdfe37fe640 1 -- 172.21.15.184:0/2460142609 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdfdc06b4a0 con 0x7fdfec0fe750 2024-06-06T07:37:02.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.178+0000 7fdfe3fff640 1 --2- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fdfd8037890 0x7fdfd8039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:02.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.178+0000 7fdfe3fff640 1 --2- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fdfd8037890 0x7fdfd8039d50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40098c0 tx=0x7fdfd4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:02.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.183+0000 7fdfe37fe640 1 -- 172.21.15.184:0/2460142609 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fdfdc04d9a0 con 0x7fdfec0fe750 2024-06-06T07:37:02.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:02.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:02.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:02.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:02.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:02.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:02.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: Regenerating cephadm self-signed grafana TLS certificates 2024-06-06T07:37:02.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:02.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:02.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-06T07:37:02.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-06T07:37:02.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:02.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: Deploying daemon grafana.smithi082 on smithi082 2024-06-06T07:37:02.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:01 smithi082 ceph-mon[29354]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:02.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.419+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfec02c3e0 con 0x7fdfec0fe750 2024-06-06T07:37:02.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.419+0000 7fdfe37fe640 1 -- 172.21.15.184:0/2460142609 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fdfdc035070 con 0x7fdfec0fe750 2024-06-06T07:37:02.421 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:02.421 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:02.421 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:02.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fdfd8037890 msgr2=0x7fdfd8039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:02.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 --2- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fdfd8037890 0x7fdfd8039d50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40098c0 tx=0x7fdfd4006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:02.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 msgr2=0x7fdfec19b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:02.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 --2- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 0x7fdfec19b9b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc006010 tx=0x7fdfdc005630 comp rx=0 tx=0).stop 2024-06-06T07:37:02.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 shutdown_connections 2024-06-06T07:37:02.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 --2- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7fdfd8037890 0x7fdfd8039d50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:02.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 --2- 172.21.15.184:0/2460142609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfec0fe750 0x7fdfec19b9b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:02.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 >> 172.21.15.184:0/2460142609 conn(0x7fdfec0fa370 msgr2=0x7fdfec0fac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:02.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 shutdown_connections 2024-06-06T07:37:02.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:02.421+0000 7fdff0f54640 1 -- 172.21.15.184:0/2460142609 wait complete. 2024-06-06T07:37:03.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:02 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/2460142609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:03.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:03.998 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:04.200 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:04.246 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:04.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:03 smithi082 ceph-mon[29354]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:05.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.589+0000 7f6bb7f90640 1 -- 172.21.15.184:0/4075739976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 msgr2=0x7f6bb00febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:05.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.589+0000 7f6bb7f90640 1 --2- 172.21.15.184:0/4075739976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 0x7f6bb00febf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6bac009880 tx=0x7f6bac02a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:05.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.589+0000 7f6bb7f90640 1 -- 172.21.15.184:0/4075739976 shutdown_connections 2024-06-06T07:37:05.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.589+0000 7f6bb7f90640 1 --2- 172.21.15.184:0/4075739976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 0x7f6bb00febf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:05.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.589+0000 7f6bb7f90640 1 -- 172.21.15.184:0/4075739976 >> 172.21.15.184:0/4075739976 conn(0x7f6bb00fa370 msgr2=0x7f6bb00fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:05.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.589+0000 7f6bb7f90640 1 -- 172.21.15.184:0/4075739976 shutdown_connections 2024-06-06T07:37:05.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.589+0000 7f6bb7f90640 1 -- 172.21.15.184:0/4075739976 wait complete. 2024-06-06T07:37:05.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.590+0000 7f6bb7f90640 1 Processor -- start 2024-06-06T07:37:05.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.590+0000 7f6bb7f90640 1 -- start start 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.590+0000 7f6bb7f90640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 0x7f6bb01975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.590+0000 7f6bb7f90640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb0197b10 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.590+0000 7f6bb5d2c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 0x7f6bb01975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.590+0000 7f6bb5d2c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 0x7f6bb01975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:34738/0 (socket says 172.21.15.184:34738) 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.590+0000 7f6bb5d2c640 1 -- 172.21.15.184:0/1034745733 learned_addr learned my addr 172.21.15.184:0/1034745733 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.590+0000 7f6bb5d2c640 1 -- 172.21.15.184:0/1034745733 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac009560 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.591+0000 7f6bb5d2c640 1 --2- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 0x7f6bb01975d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6bac006010 tx=0x7f6bac005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.591+0000 7f6ba6ffd640 1 -- 172.21.15.184:0/1034745733 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bac004aa0 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.591+0000 7f6ba6ffd640 1 -- 172.21.15.184:0/1034745733 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bac004c00 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.591+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb0197d10 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.591+0000 7f6ba6ffd640 1 -- 172.21.15.184:0/1034745733 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bac031a80 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.591+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb01981b0 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.592+0000 7f6ba6ffd640 1 -- 172.21.15.184:0/1034745733 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f6bac031cf0 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.592+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb0061ae0 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.592+0000 7f6ba6ffd640 1 --2- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6b9c0378e0 0x7f6b9c039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:05.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.592+0000 7f6ba6ffd640 1 -- 172.21.15.184:0/1034745733 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6bac06b520 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.592+0000 7f6bb552b640 1 --2- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6b9c0378e0 0x7f6b9c039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:05.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.593+0000 7f6bb552b640 1 --2- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6b9c0378e0 0x7f6b9c039da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00098c0 tx=0x7f6ba0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:05.598 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.598+0000 7f6ba6ffd640 1 -- 172.21.15.184:0/1034745733 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6bac04d530 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.835 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.833+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bb0190ed0 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.835 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.834+0000 7f6ba6ffd640 1 -- 172.21.15.184:0/1034745733 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6bac035070 con 0x7f6bb00fe7d0 2024-06-06T07:37:05.836 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:05.836 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:05.836 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:05.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6b9c0378e0 msgr2=0x7f6b9c039da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:05.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 --2- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6b9c0378e0 0x7f6b9c039da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00098c0 tx=0x7f6ba0006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:05.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 msgr2=0x7f6bb01975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:05.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 --2- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 0x7f6bb01975d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6bac006010 tx=0x7f6bac005630 comp rx=0 tx=0).stop 2024-06-06T07:37:05.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 shutdown_connections 2024-06-06T07:37:05.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 --2- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f6b9c0378e0 0x7f6b9c039da0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:05.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 --2- 172.21.15.184:0/1034745733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6bb00fe7d0 0x7f6bb01975d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:05.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 >> 172.21.15.184:0/1034745733 conn(0x7f6bb00fa370 msgr2=0x7f6bb00faee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:05.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 shutdown_connections 2024-06-06T07:37:05.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:05.836+0000 7f6bb7f90640 1 -- 172.21.15.184:0/1034745733 wait complete. 2024-06-06T07:37:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:06 smithi082 ceph-mon[29354]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:06 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:06 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/1034745733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:07.428 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:07.428 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:07.631 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:07.678 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:08.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:08 smithi082 ceph-mon[29354]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:09.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.049+0000 7f0e93fff640 1 -- 172.21.15.184:0/4083542325 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e8c0317f0 con 0x7f0e940fede0 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.050+0000 7f0e9b39a640 1 -- 172.21.15.184:0/4083542325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 msgr2=0x7f0e940ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.050+0000 7f0e9b39a640 1 --2- 172.21.15.184:0/4083542325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 0x7f0e940ff200 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009880 tx=0x7f0e8c02a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.050+0000 7f0e9b39a640 1 -- 172.21.15.184:0/4083542325 shutdown_connections 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.050+0000 7f0e9b39a640 1 --2- 172.21.15.184:0/4083542325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 0x7f0e940ff200 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.050+0000 7f0e9b39a640 1 -- 172.21.15.184:0/4083542325 >> 172.21.15.184:0/4083542325 conn(0x7f0e940fa380 msgr2=0x7f0e940fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.050+0000 7f0e9b39a640 1 -- 172.21.15.184:0/4083542325 shutdown_connections 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.050+0000 7f0e9b39a640 1 -- 172.21.15.184:0/4083542325 wait complete. 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.050+0000 7f0e9b39a640 1 Processor -- start 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.050+0000 7f0e9b39a640 1 -- start start 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e9b39a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 0x7f0e94195400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:09.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e9b39a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e94195940 con 0x7f0e940fede0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e99136640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 0x7f0e94195400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e99136640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 0x7f0e94195400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:34750/0 (socket says 172.21.15.184:34750) 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e99136640 1 -- 172.21.15.184:0/1813765243 learned_addr learned my addr 172.21.15.184:0/1813765243 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e99136640 1 -- 172.21.15.184:0/1813765243 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c009560 con 0x7f0e940fede0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e99136640 1 --2- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 0x7f0e94195400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c005610 tx=0x7f0e8c005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e927fc640 1 -- 172.21.15.184:0/1813765243 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e8c004b20 con 0x7f0e940fede0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e94195b40 con 0x7f0e940fede0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e927fc640 1 -- 172.21.15.184:0/1813765243 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e8c004c80 con 0x7f0e940fede0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.051+0000 7f0e927fc640 1 -- 172.21.15.184:0/1813765243 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e8c031a90 con 0x7f0e940fede0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.052+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e94195fe0 con 0x7f0e940fede0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.052+0000 7f0e927fc640 1 -- 172.21.15.184:0/1813765243 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f0e8c031d50 con 0x7f0e940fede0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.052+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e94061ae0 con 0x7f0e940fede0 2024-06-06T07:37:09.055 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.053+0000 7f0e927fc640 1 --2- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f0e7c037890 0x7f0e7c039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:09.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.053+0000 7f0e927fc640 1 -- 172.21.15.184:0/1813765243 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0e8c06b3d0 con 0x7f0e940fede0 2024-06-06T07:37:09.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.053+0000 7f0e98935640 1 --2- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f0e7c037890 0x7f0e7c039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:09.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.053+0000 7f0e98935640 1 --2- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f0e7c037890 0x7f0e7c039d50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800ad30 tx=0x7f0e880093f0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:09.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.058+0000 7f0e927fc640 1 -- 172.21.15.184:0/1813765243 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0e8c04dce0 con 0x7f0e940fede0 2024-06-06T07:37:09.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.297+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e9418ed00 con 0x7f0e940fede0 2024-06-06T07:37:09.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.297+0000 7f0e927fc640 1 -- 172.21.15.184:0/1813765243 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f0e8c035070 con 0x7f0e940fede0 2024-06-06T07:37:09.298 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:09.298 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:09.299 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:09.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.299+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f0e7c037890 msgr2=0x7f0e7c039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:09.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.299+0000 7f0e9b39a640 1 --2- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f0e7c037890 0x7f0e7c039d50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800ad30 tx=0x7f0e880093f0 comp rx=0 tx=0).stop 2024-06-06T07:37:09.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.299+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 msgr2=0x7f0e94195400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:09.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.299+0000 7f0e9b39a640 1 --2- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 0x7f0e94195400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c005610 tx=0x7f0e8c005640 comp rx=0 tx=0).stop 2024-06-06T07:37:09.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.300+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 shutdown_connections 2024-06-06T07:37:09.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.300+0000 7f0e9b39a640 1 --2- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f0e7c037890 0x7f0e7c039d50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:09.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.300+0000 7f0e9b39a640 1 --2- 172.21.15.184:0/1813765243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e940fede0 0x7f0e94195400 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:09.301 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.300+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 >> 172.21.15.184:0/1813765243 conn(0x7f0e940fa380 msgr2=0x7f0e940fae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:09.301 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.300+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 shutdown_connections 2024-06-06T07:37:09.301 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:09.300+0000 7f0e9b39a640 1 -- 172.21.15.184:0/1813765243 wait complete. 2024-06-06T07:37:09.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:09 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/1813765243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:10.763 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:10 smithi082 ceph-mon[29354]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:10.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:10.883 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:11.088 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:11.135 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:12.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.560+0000 7f21bfd60640 1 -- 172.21.15.184:0/1410199239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 msgr2=0x7f21b8100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:12.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.560+0000 7f21bfd60640 1 --2- 172.21.15.184:0/1410199239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 0x7f21b8100bb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009880 tx=0x7f21b402a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.560+0000 7f21bfd60640 1 -- 172.21.15.184:0/1410199239 shutdown_connections 2024-06-06T07:37:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.560+0000 7f21bfd60640 1 --2- 172.21.15.184:0/1410199239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 0x7f21b8100bb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.560+0000 7f21bfd60640 1 -- 172.21.15.184:0/1410199239 >> 172.21.15.184:0/1410199239 conn(0x7f21b80fa3a0 msgr2=0x7f21b80fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.560+0000 7f21bfd60640 1 -- 172.21.15.184:0/1410199239 shutdown_connections 2024-06-06T07:37:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.560+0000 7f21bfd60640 1 -- 172.21.15.184:0/1410199239 wait complete. 2024-06-06T07:37:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.561+0000 7f21bfd60640 1 Processor -- start 2024-06-06T07:37:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.561+0000 7f21bfd60640 1 -- start start 2024-06-06T07:37:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.561+0000 7f21bfd60640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 0x7f21b819b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.561+0000 7f21bfd60640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b819bf00 con 0x7f21b80fe780 2024-06-06T07:37:12.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.561+0000 7f21bdafc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 0x7f21b819b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.561+0000 7f21bdafc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 0x7f21b819b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:34764/0 (socket says 172.21.15.184:34764) 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.561+0000 7f21bdafc640 1 -- 172.21.15.184:0/3839126201 learned_addr learned my addr 172.21.15.184:0/3839126201 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.562+0000 7f21bdafc640 1 -- 172.21.15.184:0/3839126201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b4009560 con 0x7f21b80fe780 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.562+0000 7f21bdafc640 1 --2- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 0x7f21b819b9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f21b4005610 tx=0x7f21b4005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.562+0000 7f21aeffd640 1 -- 172.21.15.184:0/3839126201 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21b4004be0 con 0x7f21b80fe780 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.562+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b819c100 con 0x7f21b80fe780 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.562+0000 7f21aeffd640 1 -- 172.21.15.184:0/3839126201 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21b4004d40 con 0x7f21b80fe780 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.562+0000 7f21aeffd640 1 -- 172.21.15.184:0/3839126201 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21b4031b50 con 0x7f21b80fe780 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.562+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b819c5a0 con 0x7f21b80fe780 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.563+0000 7f21aeffd640 1 -- 172.21.15.184:0/3839126201 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f21b40035f0 con 0x7f21b80fe780 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.563+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b8061ae0 con 0x7f21b80fe780 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.563+0000 7f21aeffd640 1 --2- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f21a4037890 0x7f21a4039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:12.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.563+0000 7f21aeffd640 1 -- 172.21.15.184:0/3839126201 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f21b406b480 con 0x7f21b80fe780 2024-06-06T07:37:12.565 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.564+0000 7f21bd2fb640 1 --2- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f21a4037890 0x7f21a4039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:12.565 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.564+0000 7f21bd2fb640 1 --2- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f21a4037890 0x7f21a4039d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f21a80098c0 tx=0x7f21a8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:12.568 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.568+0000 7f21aeffd640 1 -- 172.21.15.184:0/3839126201 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f21b404d9a0 con 0x7f21b80fe780 2024-06-06T07:37:12.725 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:12 smithi082 ceph-mon[29354]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:12.806 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.805+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21b802c3e0 con 0x7f21b80fe780 2024-06-06T07:37:12.806 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.806+0000 7f21aeffd640 1 -- 172.21.15.184:0/3839126201 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f21b4035070 con 0x7f21b80fe780 2024-06-06T07:37:12.806 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:12.807 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:12.807 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:12.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f21a4037890 msgr2=0x7f21a4039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:12.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 --2- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f21a4037890 0x7f21a4039d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f21a80098c0 tx=0x7f21a8006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:12.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 msgr2=0x7f21b819b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:12.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 --2- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 0x7f21b819b9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f21b4005610 tx=0x7f21b4005640 comp rx=0 tx=0).stop 2024-06-06T07:37:12.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 shutdown_connections 2024-06-06T07:37:12.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 --2- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f21a4037890 0x7f21a4039d50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:12.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 --2- 172.21.15.184:0/3839126201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21b80fe780 0x7f21b819b9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:12.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 >> 172.21.15.184:0/3839126201 conn(0x7f21b80fa3a0 msgr2=0x7f21b80fac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:12.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 shutdown_connections 2024-06-06T07:37:12.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:12.808+0000 7f21bfd60640 1 -- 172.21.15.184:0/3839126201 wait complete. 2024-06-06T07:37:13.729 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:13 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/3839126201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:14.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:14.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:14.482 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:14 smithi082 ceph-mon[29354]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:14.483 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:14 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:14.483 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:14 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:14.483 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:14 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:14.483 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:14 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:14.483 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:14 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:14.483 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:14 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:14.483 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:14 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:14.483 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:14 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:14.564 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:14.610 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:15.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:15 smithi082 ceph-mon[29354]: Deploying daemon prometheus.smithi082 on smithi082 2024-06-06T07:37:15.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:15 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:16.039 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.038+0000 7f2670833640 1 -- 172.21.15.184:0/624249432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 msgr2=0x7f266c0ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:16.039 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.038+0000 7f2670833640 1 --2- 172.21.15.184:0/624249432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 0x7f266c0ff200 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f265c009880 tx=0x7f265c02a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:16.039 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.039+0000 7f2670833640 1 -- 172.21.15.184:0/624249432 shutdown_connections 2024-06-06T07:37:16.039 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.039+0000 7f2670833640 1 --2- 172.21.15.184:0/624249432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 0x7f266c0ff200 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:16.039 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.039+0000 7f2670833640 1 -- 172.21.15.184:0/624249432 >> 172.21.15.184:0/624249432 conn(0x7f266c0fa380 msgr2=0x7f266c0fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:16.040 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.039+0000 7f2670833640 1 -- 172.21.15.184:0/624249432 shutdown_connections 2024-06-06T07:37:16.040 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.039+0000 7f2670833640 1 -- 172.21.15.184:0/624249432 wait complete. 2024-06-06T07:37:16.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.039+0000 7f2670833640 1 Processor -- start 2024-06-06T07:37:16.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.039+0000 7f2670833640 1 -- start start 2024-06-06T07:37:16.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.040+0000 7f2670833640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 0x7f266c197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:16.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.040+0000 7f2670833640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c197b40 con 0x7f266c0fede0 2024-06-06T07:37:16.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.040+0000 7f266a59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 0x7f266c197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:16.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.040+0000 7f266a59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 0x7f266c197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:41684/0 (socket says 172.21.15.184:41684) 2024-06-06T07:37:16.042 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.040+0000 7f266a59c640 1 -- 172.21.15.184:0/1471489820 learned_addr learned my addr 172.21.15.184:0/1471489820 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:16.042 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.040+0000 7f266a59c640 1 -- 172.21.15.184:0/1471489820 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c009560 con 0x7f266c0fede0 2024-06-06T07:37:16.042 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.040+0000 7f266a59c640 1 --2- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 0x7f266c197600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f265c005610 tx=0x7f265c005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:16.042 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.041+0000 7f2663fff640 1 -- 172.21.15.184:0/1471489820 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f265c004b20 con 0x7f266c0fede0 2024-06-06T07:37:16.042 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.041+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c197d40 con 0x7f266c0fede0 2024-06-06T07:37:16.042 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.041+0000 7f2663fff640 1 -- 172.21.15.184:0/1471489820 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f265c004c80 con 0x7f266c0fede0 2024-06-06T07:37:16.042 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.041+0000 7f2663fff640 1 -- 172.21.15.184:0/1471489820 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f265c031a90 con 0x7f266c0fede0 2024-06-06T07:37:16.042 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.041+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c1981e0 con 0x7f266c0fede0 2024-06-06T07:37:16.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.042+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c061ae0 con 0x7f266c0fede0 2024-06-06T07:37:16.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.042+0000 7f2663fff640 1 -- 172.21.15.184:0/1471489820 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f265c031d50 con 0x7f266c0fede0 2024-06-06T07:37:16.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.042+0000 7f2663fff640 1 --2- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2658037890 0x7f2658039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:16.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.042+0000 7f2663fff640 1 -- 172.21.15.184:0/1471489820 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f265c06b440 con 0x7f266c0fede0 2024-06-06T07:37:16.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.042+0000 7f2661d9b640 1 --2- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2658037890 0x7f2658039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:16.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.042+0000 7f2661d9b640 1 --2- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2658037890 0x7f2658039d50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26540098c0 tx=0x7f2654006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:16.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.051+0000 7f2663fff640 1 -- 172.21.15.184:0/1471489820 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f265c04d9a0 con 0x7f266c0fede0 2024-06-06T07:37:16.287 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.286+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f266c02c3e0 con 0x7f266c0fede0 2024-06-06T07:37:16.287 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.287+0000 7f2663fff640 1 -- 172.21.15.184:0/1471489820 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f265c035070 con 0x7f266c0fede0 2024-06-06T07:37:16.287 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:16.288 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:16.288 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:16.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.289+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2658037890 msgr2=0x7f2658039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:16.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.289+0000 7f2670833640 1 --2- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2658037890 0x7f2658039d50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26540098c0 tx=0x7f2654006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:16.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.289+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 msgr2=0x7f266c197600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:16.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.289+0000 7f2670833640 1 --2- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 0x7f266c197600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f265c005610 tx=0x7f265c005640 comp rx=0 tx=0).stop 2024-06-06T07:37:16.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.289+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 shutdown_connections 2024-06-06T07:37:16.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.289+0000 7f2670833640 1 --2- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2658037890 0x7f2658039d50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:16.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.289+0000 7f2670833640 1 --2- 172.21.15.184:0/1471489820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f266c0fede0 0x7f266c197600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:16.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.290+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 >> 172.21.15.184:0/1471489820 conn(0x7f266c0fa380 msgr2=0x7f266c18e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:16.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.290+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 shutdown_connections 2024-06-06T07:37:16.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:16.290+0000 7f2670833640 1 -- 172.21.15.184:0/1471489820 wait complete. 2024-06-06T07:37:16.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:16 smithi082 ceph-mon[29354]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:16.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:16 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/1471489820' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:17.887 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:17.887 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:18.085 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:18.132 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:18.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:18 smithi082 ceph-mon[29354]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:19.519 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.519+0000 7ff504bbe640 1 -- 172.21.15.184:0/3284227946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 msgr2=0x7ff500102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:19.519 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.519+0000 7ff504bbe640 1 --2- 172.21.15.184:0/3284227946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 0x7ff500102d70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009880 tx=0x7ff4f002a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:19.519 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.519+0000 7ff504bbe640 1 -- 172.21.15.184:0/3284227946 shutdown_connections 2024-06-06T07:37:19.519 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.519+0000 7ff504bbe640 1 --2- 172.21.15.184:0/3284227946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 0x7ff500102d70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:19.520 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.519+0000 7ff504bbe640 1 -- 172.21.15.184:0/3284227946 >> 172.21.15.184:0/3284227946 conn(0x7ff5000fa3b0 msgr2=0x7ff5000fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:19.520 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.519+0000 7ff504bbe640 1 -- 172.21.15.184:0/3284227946 shutdown_connections 2024-06-06T07:37:19.520 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.519+0000 7ff504bbe640 1 -- 172.21.15.184:0/3284227946 wait complete. 2024-06-06T07:37:19.520 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.520+0000 7ff504bbe640 1 Processor -- start 2024-06-06T07:37:19.520 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.520+0000 7ff504bbe640 1 -- start start 2024-06-06T07:37:19.520 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.520+0000 7ff504bbe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 0x7ff500195380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:19.521 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.520+0000 7ff504bbe640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5001958c0 con 0x7ff500100980 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.520+0000 7ff4fe59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 0x7ff500195380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.520+0000 7ff4fe59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 0x7ff500195380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:41710/0 (socket says 172.21.15.184:41710) 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.520+0000 7ff4fe59c640 1 -- 172.21.15.184:0/2645696249 learned_addr learned my addr 172.21.15.184:0/2645696249 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.521+0000 7ff4fe59c640 1 -- 172.21.15.184:0/2645696249 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f0009560 con 0x7ff500100980 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.521+0000 7ff4fe59c640 1 --2- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 0x7ff500195380 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0006010 tx=0x7ff4f0005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.521+0000 7ff4f7fff640 1 -- 172.21.15.184:0/2645696249 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f0004aa0 con 0x7ff500100980 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.521+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff500195ac0 con 0x7ff500100980 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.521+0000 7ff4f7fff640 1 -- 172.21.15.184:0/2645696249 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4f0004c00 con 0x7ff500100980 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.521+0000 7ff4f7fff640 1 -- 172.21.15.184:0/2645696249 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4f0031a80 con 0x7ff500100980 2024-06-06T07:37:19.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.521+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff500195f60 con 0x7ff500100980 2024-06-06T07:37:19.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.522+0000 7ff4f7fff640 1 -- 172.21.15.184:0/2645696249 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7ff4f0031d40 con 0x7ff500100980 2024-06-06T07:37:19.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.522+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff500061ae0 con 0x7ff500100980 2024-06-06T07:37:19.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.522+0000 7ff4f7fff640 1 --2- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff4ec037890 0x7ff4ec039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:19.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.522+0000 7ff4f7fff640 1 -- 172.21.15.184:0/2645696249 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff4f006b4a0 con 0x7ff500100980 2024-06-06T07:37:19.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.523+0000 7ff4f5d9b640 1 --2- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff4ec037890 0x7ff4ec039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:19.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.523+0000 7ff4f5d9b640 1 --2- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff4ec037890 0x7ff4ec039d50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff4e80098c0 tx=0x7ff4e8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:19.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.528+0000 7ff4f7fff640 1 -- 172.21.15.184:0/2645696249 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff4f004d9a0 con 0x7ff500100980 2024-06-06T07:37:19.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.761+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff50002c3e0 con 0x7ff500100980 2024-06-06T07:37:19.762 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.762+0000 7ff4f7fff640 1 -- 172.21.15.184:0/2645696249 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff4f0035070 con 0x7ff500100980 2024-06-06T07:37:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:19.763 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:19.763 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:19.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff4ec037890 msgr2=0x7ff4ec039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:19.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 --2- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff4ec037890 0x7ff4ec039d50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff4e80098c0 tx=0x7ff4e8006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:19.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 msgr2=0x7ff500195380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:19.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 --2- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 0x7ff500195380 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0006010 tx=0x7ff4f0005630 comp rx=0 tx=0).stop 2024-06-06T07:37:19.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 shutdown_connections 2024-06-06T07:37:19.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 --2- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff4ec037890 0x7ff4ec039d50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:19.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 --2- 172.21.15.184:0/2645696249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff500100980 0x7ff500195380 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:19.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 >> 172.21.15.184:0/2645696249 conn(0x7ff5000fa3b0 msgr2=0x7ff5000fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:19.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 shutdown_connections 2024-06-06T07:37:19.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:19.764+0000 7ff504bbe640 1 -- 172.21.15.184:0/2645696249 wait complete. 2024-06-06T07:37:20.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:20 smithi082 ceph-mon[29354]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:20.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:20 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/2645696249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:21.341 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:21.341 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:21.541 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:21.588 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:22.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:22 smithi082 ceph-mon[29354]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:22.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.955+0000 7f812d0ef640 1 -- 172.21.15.184:0/2015583675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 msgr2=0x7f81280ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:22.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.955+0000 7f812d0ef640 1 --2- 172.21.15.184:0/2015583675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 0x7f81280ff200 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8110009880 tx=0x7f811002a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:22.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.955+0000 7f812d0ef640 1 -- 172.21.15.184:0/2015583675 shutdown_connections 2024-06-06T07:37:22.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.955+0000 7f812d0ef640 1 --2- 172.21.15.184:0/2015583675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 0x7f81280ff200 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:22.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.955+0000 7f812d0ef640 1 -- 172.21.15.184:0/2015583675 >> 172.21.15.184:0/2015583675 conn(0x7f81280fa380 msgr2=0x7f81280fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:22.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.955+0000 7f812d0ef640 1 -- 172.21.15.184:0/2015583675 shutdown_connections 2024-06-06T07:37:22.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.955+0000 7f812d0ef640 1 -- 172.21.15.184:0/2015583675 wait complete. 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.956+0000 7f812d0ef640 1 Processor -- start 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.956+0000 7f812d0ef640 1 -- start start 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.956+0000 7f812d0ef640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 0x7f8128195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.956+0000 7f812d0ef640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8128195990 con 0x7f81280fede0 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f8126d9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 0x7f8128195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f8126d9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 0x7f8128195450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:41726/0 (socket says 172.21.15.184:41726) 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f8126d9d640 1 -- 172.21.15.184:0/2816014577 learned_addr learned my addr 172.21.15.184:0/2816014577 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f8126d9d640 1 -- 172.21.15.184:0/2816014577 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8110009560 con 0x7f81280fede0 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f8126d9d640 1 --2- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 0x7f8128195450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8110005610 tx=0x7f8110005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f811ffff640 1 -- 172.21.15.184:0/2816014577 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8110004b20 con 0x7f81280fede0 2024-06-06T07:37:22.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8128195b90 con 0x7f81280fede0 2024-06-06T07:37:22.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f811ffff640 1 -- 172.21.15.184:0/2816014577 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8110004c80 con 0x7f81280fede0 2024-06-06T07:37:22.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f811ffff640 1 -- 172.21.15.184:0/2816014577 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8110031a90 con 0x7f81280fede0 2024-06-06T07:37:22.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.957+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8128196030 con 0x7f81280fede0 2024-06-06T07:37:22.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.959+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8128061ae0 con 0x7f81280fede0 2024-06-06T07:37:22.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.959+0000 7f811ffff640 1 -- 172.21.15.184:0/2816014577 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f8110031d50 con 0x7f81280fede0 2024-06-06T07:37:22.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.959+0000 7f811ffff640 1 --2- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f8114037890 0x7f8114039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:22.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.959+0000 7f811ffff640 1 -- 172.21.15.184:0/2816014577 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f811006b340 con 0x7f81280fede0 2024-06-06T07:37:22.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.959+0000 7f812659c640 1 --2- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f8114037890 0x7f8114039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:22.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.959+0000 7f812659c640 1 --2- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f8114037890 0x7f8114039d50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f81180098c0 tx=0x7f8118006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:22.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:22.964+0000 7f811ffff640 1 -- 172.21.15.184:0/2816014577 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f81100443b0 con 0x7f81280fede0 2024-06-06T07:37:23.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.206+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f812818eed0 con 0x7f81280fede0 2024-06-06T07:37:23.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.207+0000 7f811ffff640 1 -- 172.21.15.184:0/2816014577 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f8110035070 con 0x7f81280fede0 2024-06-06T07:37:23.208 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:23.208 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:23.208 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:23.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f8114037890 msgr2=0x7f8114039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:23.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 --2- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f8114037890 0x7f8114039d50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f81180098c0 tx=0x7f8118006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:23.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 msgr2=0x7f8128195450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:23.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 --2- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 0x7f8128195450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8110005610 tx=0x7f8110005640 comp rx=0 tx=0).stop 2024-06-06T07:37:23.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 shutdown_connections 2024-06-06T07:37:23.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 --2- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f8114037890 0x7f8114039d50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:23.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 --2- 172.21.15.184:0/2816014577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81280fede0 0x7f8128195450 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:23.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 >> 172.21.15.184:0/2816014577 conn(0x7f81280fa380 msgr2=0x7f8128067b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:23.211 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 shutdown_connections 2024-06-06T07:37:23.211 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:23.209+0000 7f812d0ef640 1 -- 172.21.15.184:0/2816014577 wait complete. 2024-06-06T07:37:23.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:23 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/2816014577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:24.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:24 smithi082 ceph-mon[29354]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:24.778 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:24.779 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:24.979 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:25.026 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:26.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.464+0000 7f96419bb640 1 -- 172.21.15.184:0/2433079889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 msgr2=0x7f963c1002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:26.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.464+0000 7f96419bb640 1 --2- 172.21.15.184:0/2433079889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 0x7f963c1002b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9634009c20 tx=0x7f963402aa90 comp rx=0 tx=0).stop 2024-06-06T07:37:26.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.464+0000 7f96419bb640 1 -- 172.21.15.184:0/2433079889 shutdown_connections 2024-06-06T07:37:26.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.464+0000 7f96419bb640 1 --2- 172.21.15.184:0/2433079889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 0x7f963c1002b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:26.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.464+0000 7f96419bb640 1 -- 172.21.15.184:0/2433079889 >> 172.21.15.184:0/2433079889 conn(0x7f963c0fb480 msgr2=0x7f963c0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:26.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.465+0000 7f96419bb640 1 -- 172.21.15.184:0/2433079889 shutdown_connections 2024-06-06T07:37:26.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.465+0000 7f96419bb640 1 -- 172.21.15.184:0/2433079889 wait complete. 2024-06-06T07:37:26.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.465+0000 7f96419bb640 1 Processor -- start 2024-06-06T07:37:26.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.465+0000 7f96419bb640 1 -- start start 2024-06-06T07:37:26.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.466+0000 7f96419bb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 0x7f963c074120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:26.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.466+0000 7f96419bb640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963c074660 con 0x7f963c0ffe90 2024-06-06T07:37:26.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.466+0000 7f963affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 0x7f963c074120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:26.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.466+0000 7f963affd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 0x7f963c074120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:58384/0 (socket says 172.21.15.184:58384) 2024-06-06T07:37:26.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.466+0000 7f963affd640 1 -- 172.21.15.184:0/3021286188 learned_addr learned my addr 172.21.15.184:0/3021286188 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:26.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.466+0000 7f963affd640 1 -- 172.21.15.184:0/3021286188 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9634009900 con 0x7f963c0ffe90 2024-06-06T07:37:26.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.467+0000 7f963affd640 1 --2- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 0x7f963c074120 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9634005cc0 tx=0x7f963402e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:26.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.467+0000 7f96409b9640 1 -- 172.21.15.184:0/3021286188 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f963403e070 con 0x7f963c0ffe90 2024-06-06T07:37:26.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.467+0000 7f96409b9640 1 -- 172.21.15.184:0/3021286188 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f963402e990 con 0x7f963c0ffe90 2024-06-06T07:37:26.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.467+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963c0728a0 con 0x7f963c0ffe90 2024-06-06T07:37:26.475 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.467+0000 7f96409b9640 1 -- 172.21.15.184:0/3021286188 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9634038450 con 0x7f963c0ffe90 2024-06-06T07:37:26.475 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.467+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963c072d40 con 0x7f963c0ffe90 2024-06-06T07:37:26.475 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.468+0000 7f96409b9640 1 -- 172.21.15.184:0/3021286188 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f963400a040 con 0x7f963c0ffe90 2024-06-06T07:37:26.475 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.468+0000 7f96409b9640 1 --2- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f9624037890 0x7f9624039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:26.476 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.468+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963c106610 con 0x7f963c0ffe90 2024-06-06T07:37:26.476 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.468+0000 7f96409b9640 1 -- 172.21.15.184:0/3021286188 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f963406b270 con 0x7f963c0ffe90 2024-06-06T07:37:26.476 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.468+0000 7f963a7fc640 1 --2- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f9624037890 0x7f9624039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:26.476 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.469+0000 7f963a7fc640 1 --2- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f9624037890 0x7f9624039d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f96300098c0 tx=0x7f9630006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:26.476 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.473+0000 7f96409b9640 1 -- 172.21.15.184:0/3021286188 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9634047080 con 0x7f963c0ffe90 2024-06-06T07:37:26.711 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.711+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f963c105cf0 con 0x7f963c0ffe90 2024-06-06T07:37:26.712 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.711+0000 7f96409b9640 1 -- 172.21.15.184:0/3021286188 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9634033020 con 0x7f963c0ffe90 2024-06-06T07:37:26.712 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:26.712 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:26.713 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:26.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.713+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f9624037890 msgr2=0x7f9624039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:26.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.713+0000 7f96419bb640 1 --2- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f9624037890 0x7f9624039d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f96300098c0 tx=0x7f9630006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:26.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.713+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 msgr2=0x7f963c074120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:26.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.713+0000 7f96419bb640 1 --2- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 0x7f963c074120 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9634005cc0 tx=0x7f963402e5f0 comp rx=0 tx=0).stop 2024-06-06T07:37:26.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.713+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 shutdown_connections 2024-06-06T07:37:26.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.713+0000 7f96419bb640 1 --2- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f9624037890 0x7f9624039d50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:26.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.713+0000 7f96419bb640 1 --2- 172.21.15.184:0/3021286188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f963c0ffe90 0x7f963c074120 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:26.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.713+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 >> 172.21.15.184:0/3021286188 conn(0x7f963c0fb480 msgr2=0x7f963c0fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:26.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.713+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 shutdown_connections 2024-06-06T07:37:26.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:26.714+0000 7f96419bb640 1 -- 172.21.15.184:0/3021286188 wait complete. 2024-06-06T07:37:26.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:26 smithi082 ceph-mon[29354]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:27.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:27 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/3021286188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:28.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:28.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:28.420 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:28.467 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:28.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:28 smithi082 ceph-mon[29354]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:29.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.880+0000 7ff3c9767640 1 -- 172.21.15.184:0/1413373118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 msgr2=0x7ff3c40ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:29.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.880+0000 7ff3c9767640 1 --2- 172.21.15.184:0/1413373118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 0x7ff3c40ff200 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009880 tx=0x7ff3b402a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:29.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.880+0000 7ff3c9767640 1 -- 172.21.15.184:0/1413373118 shutdown_connections 2024-06-06T07:37:29.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.880+0000 7ff3c9767640 1 --2- 172.21.15.184:0/1413373118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 0x7ff3c40ff200 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:29.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.880+0000 7ff3c9767640 1 -- 172.21.15.184:0/1413373118 >> 172.21.15.184:0/1413373118 conn(0x7ff3c40fa380 msgr2=0x7ff3c40fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:29.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.880+0000 7ff3c9767640 1 -- 172.21.15.184:0/1413373118 shutdown_connections 2024-06-06T07:37:29.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.880+0000 7ff3c9767640 1 -- 172.21.15.184:0/1413373118 wait complete. 2024-06-06T07:37:29.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.880+0000 7ff3c9767640 1 Processor -- start 2024-06-06T07:37:29.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.880+0000 7ff3c9767640 1 -- start start 2024-06-06T07:37:29.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c9767640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 0x7ff3c4195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:29.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c9767640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4195990 con 0x7ff3c40fede0 2024-06-06T07:37:29.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 0x7ff3c4195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:29.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 0x7ff3c4195450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:58412/0 (socket says 172.21.15.184:58412) 2024-06-06T07:37:29.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c2ffd640 1 -- 172.21.15.184:0/3547933971 learned_addr learned my addr 172.21.15.184:0/3547933971 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:29.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c2ffd640 1 -- 172.21.15.184:0/3547933971 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4009560 con 0x7ff3c40fede0 2024-06-06T07:37:29.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c2ffd640 1 --2- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 0x7ff3c4195450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4005610 tx=0x7ff3b4005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:29.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c0ff9640 1 -- 172.21.15.184:0/3547933971 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3b4004b20 con 0x7ff3c40fede0 2024-06-06T07:37:29.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c4195b90 con 0x7ff3c40fede0 2024-06-06T07:37:29.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.881+0000 7ff3c0ff9640 1 -- 172.21.15.184:0/3547933971 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3b4004c80 con 0x7ff3c40fede0 2024-06-06T07:37:29.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.882+0000 7ff3c0ff9640 1 -- 172.21.15.184:0/3547933971 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3b4031a90 con 0x7ff3c40fede0 2024-06-06T07:37:29.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.882+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c4196030 con 0x7ff3c40fede0 2024-06-06T07:37:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.882+0000 7ff3c0ff9640 1 -- 172.21.15.184:0/3547933971 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7ff3b4031d50 con 0x7ff3c40fede0 2024-06-06T07:37:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.882+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c4061ae0 con 0x7ff3c40fede0 2024-06-06T07:37:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.883+0000 7ff3c0ff9640 1 --2- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff3b0037890 0x7ff3b0039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.883+0000 7ff3c0ff9640 1 -- 172.21.15.184:0/3547933971 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff3b406b430 con 0x7ff3c40fede0 2024-06-06T07:37:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.883+0000 7ff3ba7fc640 1 --2- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff3b0037890 0x7ff3b0039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.883+0000 7ff3ba7fc640 1 --2- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff3b0037890 0x7ff3b0039d50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0098c0 tx=0x7ff3ac006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:29.888 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:29.888+0000 7ff3c0ff9640 1 -- 172.21.15.184:0/3547933971 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff3b404d9a0 con 0x7ff3c40fede0 2024-06-06T07:37:30.129 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.129+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3c402c3e0 con 0x7ff3c40fede0 2024-06-06T07:37:30.130 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.129+0000 7ff3c0ff9640 1 -- 172.21.15.184:0/3547933971 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff3b4035070 con 0x7ff3c40fede0 2024-06-06T07:37:30.130 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:30.130 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:30.130 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff3b0037890 msgr2=0x7ff3b0039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 --2- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff3b0037890 0x7ff3b0039d50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0098c0 tx=0x7ff3ac006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 msgr2=0x7ff3c4195450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 --2- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 0x7ff3c4195450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4005610 tx=0x7ff3b4005640 comp rx=0 tx=0).stop 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 shutdown_connections 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 --2- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7ff3b0037890 0x7ff3b0039d50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 --2- 172.21.15.184:0/3547933971 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3c40fede0 0x7ff3c4195450 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 >> 172.21.15.184:0/3547933971 conn(0x7ff3c40fa380 msgr2=0x7ff3c4067b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 shutdown_connections 2024-06-06T07:37:30.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:30.131+0000 7ff3c9767640 1 -- 172.21.15.184:0/3547933971 wait complete. 2024-06-06T07:37:30.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:30 smithi082 ceph-mon[29354]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:30.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:30 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/3547933971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:31.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:31.611 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:31.813 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:31.860 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:32.711 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:32 smithi082 ceph-mon[29354]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:32.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:32 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:32.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:32 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:32.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:32 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:32.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:32 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-06T07:37:33.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.283+0000 7f71fd72a640 1 -- 172.21.15.184:0/2808205986 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 msgr2=0x7f71f8102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:33.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.283+0000 7f71fd72a640 1 --2- 172.21.15.184:0/2808205986 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 0x7f71f8102d80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f71ec009880 tx=0x7f71ec02a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:33.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.284+0000 7f71fd72a640 1 -- 172.21.15.184:0/2808205986 shutdown_connections 2024-06-06T07:37:33.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.284+0000 7f71fd72a640 1 --2- 172.21.15.184:0/2808205986 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 0x7f71f8102d80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:33.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.284+0000 7f71fd72a640 1 -- 172.21.15.184:0/2808205986 >> 172.21.15.184:0/2808205986 conn(0x7f71f80fa3a0 msgr2=0x7f71f80fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.284+0000 7f71fd72a640 1 -- 172.21.15.184:0/2808205986 shutdown_connections 2024-06-06T07:37:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.284+0000 7f71fd72a640 1 -- 172.21.15.184:0/2808205986 wait complete. 2024-06-06T07:37:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.284+0000 7f71fd72a640 1 Processor -- start 2024-06-06T07:37:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.284+0000 7f71fd72a640 1 -- start start 2024-06-06T07:37:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.284+0000 7f71fd72a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 0x7f71f8100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.284+0000 7f71fd72a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f81006b0 con 0x7f71f8100990 2024-06-06T07:37:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71f6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 0x7f71f8100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71f6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 0x7f71f8100170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:41718/0 (socket says 172.21.15.184:41718) 2024-06-06T07:37:33.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71f6ffd640 1 -- 172.21.15.184:0/709400456 learned_addr learned my addr 172.21.15.184:0/709400456 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:33.287 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71f6ffd640 1 -- 172.21.15.184:0/709400456 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec009560 con 0x7f71f8100990 2024-06-06T07:37:33.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71f6ffd640 1 --2- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 0x7f71f8100170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f71ec005610 tx=0x7f71ec005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:33.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71dffff640 1 -- 172.21.15.184:0/709400456 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71ec004be0 con 0x7f71f8100990 2024-06-06T07:37:33.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71dffff640 1 -- 172.21.15.184:0/709400456 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71ec004d40 con 0x7f71f8100990 2024-06-06T07:37:33.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f80fe940 con 0x7f71f8100990 2024-06-06T07:37:33.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71dffff640 1 -- 172.21.15.184:0/709400456 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71ec031b50 con 0x7f71f8100990 2024-06-06T07:37:33.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.285+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f80fede0 con 0x7f71f8100990 2024-06-06T07:37:33.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.286+0000 7f71dffff640 1 -- 172.21.15.184:0/709400456 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 14) v1 ==== 44936+0+0 (secure 0 0 0) 0x7f71ec03c470 con 0x7f71f8100990 2024-06-06T07:37:33.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.286+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f80ff7e0 con 0x7f71f8100990 2024-06-06T07:37:33.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.286+0000 7f71dffff640 1 --2- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f71e00378e0 0x7f71e0039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:33.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.287+0000 7f71dffff640 1 -- 172.21.15.184:0/709400456 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f71ec06b5c0 con 0x7f71f8100990 2024-06-06T07:37:33.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.287+0000 7f71f67fc640 1 -- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f71e00378e0 msgr2=0x7f71e0039da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/493557010 2024-06-06T07:37:33.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.287+0000 7f71f67fc640 1 --2- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f71e00378e0 0x7f71e0039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T07:37:33.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.291+0000 7f71dffff640 1 -- 172.21.15.184:0/709400456 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f71ec04ddb0 con 0x7f71f8100990 2024-06-06T07:37:33.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.488+0000 7f71f67fc640 1 -- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f71e00378e0 msgr2=0x7f71e0039da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/493557010 2024-06-06T07:37:33.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.488+0000 7f71f67fc640 1 --2- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f71e00378e0 0x7f71e0039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T07:37:33.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.533+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71f81066c0 con 0x7f71f8100990 2024-06-06T07:37:33.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.534+0000 7f71dffff640 1 -- 172.21.15.184:0/709400456 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f71ec035070 con 0x7f71f8100990 2024-06-06T07:37:33.535 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:33.535 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:33.535 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:33.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f71e00378e0 msgr2=0x7f71e0039da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T07:37:33.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 --2- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f71e00378e0 0x7f71e0039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:33.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 msgr2=0x7f71f8100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:33.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 --2- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 0x7f71f8100170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f71ec005610 tx=0x7f71ec005640 comp rx=0 tx=0).stop 2024-06-06T07:37:33.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 shutdown_connections 2024-06-06T07:37:33.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 --2- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f71e00378e0 0x7f71e0039da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:33.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 --2- 172.21.15.184:0/709400456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f71f8100990 0x7f71f8100170 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:33.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 >> 172.21.15.184:0/709400456 conn(0x7f71f80fa3a0 msgr2=0x7f71f80fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:33.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 shutdown_connections 2024-06-06T07:37:33.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:33.537+0000 7f71fd72a640 1 -- 172.21.15.184:0/709400456 wait complete. 2024-06-06T07:37:34.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:33 smithi082 ceph-mon[29354]: from='mgr.14160 172.21.15.82:0/2559890932' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-06T07:37:34.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:33 smithi082 ceph-mon[29354]: mgrmap e14: smithi082.kocfig(active, since 67s) 2024-06-06T07:37:34.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:33 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/709400456' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:35.122 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:35.122 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:35.343 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:35.390 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:36.798 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.796+0000 7f2bf6abc640 1 -- 172.21.15.184:0/4109573973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 msgr2=0x7f2bf01002a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.796+0000 7f2bf6abc640 1 --2- 172.21.15.184:0/4109573973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 0x7f2bf01002a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009880 tx=0x7f2be402a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.796+0000 7f2bf6abc640 1 -- 172.21.15.184:0/4109573973 shutdown_connections 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.796+0000 7f2bf6abc640 1 --2- 172.21.15.184:0/4109573973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 0x7f2bf01002a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.796+0000 7f2bf6abc640 1 -- 172.21.15.184:0/4109573973 >> 172.21.15.184:0/4109573973 conn(0x7f2bf00fb490 msgr2=0x7f2bf00fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.796+0000 7f2bf6abc640 1 -- 172.21.15.184:0/4109573973 shutdown_connections 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.796+0000 7f2bf6abc640 1 -- 172.21.15.184:0/4109573973 wait complete. 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.797+0000 7f2bf6abc640 1 Processor -- start 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.797+0000 7f2bf6abc640 1 -- start start 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.797+0000 7f2bf6abc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 0x7f2bf01953c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.797+0000 7f2bf6abc640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf0195900 con 0x7f2bf00ffe80 2024-06-06T07:37:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.797+0000 7f2bf4858640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 0x7f2bf01953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:36.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.797+0000 7f2bf4858640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 0x7f2bf01953c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:41736/0 (socket says 172.21.15.184:41736) 2024-06-06T07:37:36.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.797+0000 7f2bf4858640 1 -- 172.21.15.184:0/1377244054 learned_addr learned my addr 172.21.15.184:0/1377244054 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:36.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.798+0000 7f2bf4858640 1 -- 172.21.15.184:0/1377244054 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be4009560 con 0x7f2bf00ffe80 2024-06-06T07:37:36.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.798+0000 7f2bf4858640 1 --2- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 0x7f2bf01953c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2be4005610 tx=0x7f2be4005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:36.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.798+0000 7f2bedffb640 1 -- 172.21.15.184:0/1377244054 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2be4004b20 con 0x7f2bf00ffe80 2024-06-06T07:37:36.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.798+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf0195b00 con 0x7f2bf00ffe80 2024-06-06T07:37:36.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.798+0000 7f2bedffb640 1 -- 172.21.15.184:0/1377244054 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2be4004c80 con 0x7f2bf00ffe80 2024-06-06T07:37:36.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.798+0000 7f2bedffb640 1 -- 172.21.15.184:0/1377244054 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2be4031a90 con 0x7f2bf00ffe80 2024-06-06T07:37:36.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.798+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf0195fa0 con 0x7f2bf00ffe80 2024-06-06T07:37:36.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.799+0000 7f2bedffb640 1 -- 172.21.15.184:0/1377244054 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 14) v1 ==== 44936+0+0 (secure 0 0 0) 0x7f2be4031d50 con 0x7f2bf00ffe80 2024-06-06T07:37:36.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.799+0000 7f2bedffb640 1 --2- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2be00378e0 0x7f2be0039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:36.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.799+0000 7f2bedffb640 1 -- 172.21.15.184:0/1377244054 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2be406b4d0 con 0x7f2bf00ffe80 2024-06-06T07:37:36.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.799+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf0061ae0 con 0x7f2bf00ffe80 2024-06-06T07:37:36.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.799+0000 7f2beffff640 1 -- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2be00378e0 msgr2=0x7f2be0039da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/493557010 2024-06-06T07:37:36.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.799+0000 7f2beffff640 1 --2- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2be00378e0 0x7f2be0039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T07:37:36.804 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:36.804+0000 7f2bedffb640 1 -- 172.21.15.184:0/1377244054 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2be4035070 con 0x7f2bf00ffe80 2024-06-06T07:37:37.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.000+0000 7f2beffff640 1 -- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2be00378e0 msgr2=0x7f2be0039da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.82:6800/493557010 2024-06-06T07:37:37.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.000+0000 7f2beffff640 1 --2- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2be00378e0 0x7f2be0039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T07:37:37.040 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.040+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bf018ec40 con 0x7f2bf00ffe80 2024-06-06T07:37:37.040 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.040+0000 7f2bedffb640 1 -- 172.21.15.184:0/1377244054 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f2be4054090 con 0x7f2bf00ffe80 2024-06-06T07:37:37.041 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:37.041 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:37.041 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2be00378e0 msgr2=0x7f2be0039da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 --2- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2be00378e0 0x7f2be0039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 msgr2=0x7f2bf01953c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 --2- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 0x7f2bf01953c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2be4005610 tx=0x7f2be4005640 comp rx=0 tx=0).stop 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 shutdown_connections 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 --2- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:6800/493557010,v1:172.21.15.82:6801/493557010] conn(0x7f2be00378e0 0x7f2be0039da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 --2- 172.21.15.184:0/1377244054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2bf00ffe80 0x7f2bf01953c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 >> 172.21.15.184:0/1377244054 conn(0x7f2bf00fb490 msgr2=0x7f2bf00fbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 shutdown_connections 2024-06-06T07:37:37.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:37.042+0000 7f2bf6abc640 1 -- 172.21.15.184:0/1377244054 wait complete. 2024-06-06T07:37:37.499 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:37 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/1377244054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:38.092 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: Active manager daemon smithi082.kocfig restarted 2024-06-06T07:37:38.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: Activating manager daemon smithi082.kocfig 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: osdmap e5: 0 total, 0 up, 0 in 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: mgrmap e15: smithi082.kocfig(active, starting, since 0.00362669s) 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr metadata", "who": "smithi082.kocfig", "id": "smithi082.kocfig"}]: dispatch 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: Manager daemon smithi082.kocfig is now available 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:37:38.409 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:38.410 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:38.410 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:37:38.410 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi082.kocfig/mirror_snapshot_schedule"}]: dispatch 2024-06-06T07:37:38.410 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi082.kocfig/trash_purge_schedule"}]: dispatch 2024-06-06T07:37:38.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:38.558 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:38.753 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:38.801 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:39.406 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:39 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:39.406 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:39 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:39.406 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:39 smithi082 ceph-mon[29354]: [06/Jun/2024:07:37:38] ENGINE Bus STARTING 2024-06-06T07:37:39.406 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:39 smithi082 ceph-mon[29354]: mgrmap e16: smithi082.kocfig(active, since 1.00586s) 2024-06-06T07:37:39.406 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:39 smithi082 ceph-mon[29354]: [06/Jun/2024:07:37:38] ENGINE Serving on http://172.21.15.82:8765 2024-06-06T07:37:39.406 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:39 smithi082 ceph-mon[29354]: [06/Jun/2024:07:37:38] ENGINE Serving on https://172.21.15.82:7150 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:39 smithi082 ceph-mon[29354]: [06/Jun/2024:07:37:38] ENGINE Bus STARTED 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:39 smithi082 ceph-mon[29354]: [06/Jun/2024:07:37:38] ENGINE Error in HTTPServer.serve 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: Traceback (most recent call last): 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self._connections.run(self.expiration_interval) 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self._run(expiration_interval) 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-06T07:37:39.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: s = self.context.wrap_socket( 2024-06-06T07:37:39.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-06T07:37:39.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: return self.sslsocket_class._create( 2024-06-06T07:37:39.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-06T07:37:39.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self.do_handshake() 2024-06-06T07:37:39.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-06T07:37:39.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: self._sslobj.do_handshake() 2024-06-06T07:37:39.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-06T07:37:40.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 -- 172.21.15.184:0/1939636904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 msgr2=0x7f2e40100320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:40.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 --2- 172.21.15.184:0/1939636904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 0x7f2e40100320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009880 tx=0x7f2e3002a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:40.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 -- 172.21.15.184:0/1939636904 shutdown_connections 2024-06-06T07:37:40.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 --2- 172.21.15.184:0/1939636904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 0x7f2e40100320 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:40.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 -- 172.21.15.184:0/1939636904 >> 172.21.15.184:0/1939636904 conn(0x7f2e400fb4a0 msgr2=0x7f2e400fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 -- 172.21.15.184:0/1939636904 shutdown_connections 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 -- 172.21.15.184:0/1939636904 wait complete. 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 Processor -- start 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 -- start start 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 0x7f2e40195430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.159+0000 7f2e461be640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e40195970 con 0x7f2e400fff00 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e3f7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 0x7f2e40195430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e3f7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 0x7f2e40195430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:41756/0 (socket says 172.21.15.184:41756) 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e3f7fe640 1 -- 172.21.15.184:0/2598073446 learned_addr learned my addr 172.21.15.184:0/2598073446 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e3f7fe640 1 -- 172.21.15.184:0/2598073446 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e30009560 con 0x7f2e400fff00 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e3f7fe640 1 --2- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 0x7f2e40195430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2e30005610 tx=0x7f2e30005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e3d7fa640 1 -- 172.21.15.184:0/2598073446 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e30004be0 con 0x7f2e400fff00 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e40195b70 con 0x7f2e400fff00 2024-06-06T07:37:40.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e3d7fa640 1 -- 172.21.15.184:0/2598073446 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e30004d40 con 0x7f2e400fff00 2024-06-06T07:37:40.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e3d7fa640 1 -- 172.21.15.184:0/2598073446 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e30031b50 con 0x7f2e400fff00 2024-06-06T07:37:40.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.160+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e40196010 con 0x7f2e400fff00 2024-06-06T07:37:40.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.161+0000 7f2e3d7fa640 1 -- 172.21.15.184:0/2598073446 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7f2e300035f0 con 0x7f2e400fff00 2024-06-06T07:37:40.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.161+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e40061ae0 con 0x7f2e400fff00 2024-06-06T07:37:40.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.161+0000 7f2e3d7fa640 1 --2- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2e2c037980 0x7f2e2c039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:40.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.162+0000 7f2e3d7fa640 1 -- 172.21.15.184:0/2598073446 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f2e3006b570 con 0x7f2e400fff00 2024-06-06T07:37:40.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.162+0000 7f2e37fff640 1 --2- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2e2c037980 0x7f2e2c039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:40.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.162+0000 7f2e37fff640 1 --2- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2e2c037980 0x7f2e2c039e40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2e280098c0 tx=0x7f2e28006db0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:40.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.166+0000 7f2e3d7fa640 1 -- 172.21.15.184:0/2598073446 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2e3004dcd0 con 0x7f2e400fff00 2024-06-06T07:37:40.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.420+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e4002c3e0 con 0x7f2e400fff00 2024-06-06T07:37:40.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.421+0000 7f2e3d7fa640 1 -- 172.21.15.184:0/2598073446 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f2e30035020 con 0x7f2e400fff00 2024-06-06T07:37:40.422 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:40.422 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:40.422 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:40.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2e2c037980 msgr2=0x7f2e2c039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:40.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 --2- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2e2c037980 0x7f2e2c039e40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2e280098c0 tx=0x7f2e28006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:40.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 msgr2=0x7f2e40195430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:40.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 --2- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 0x7f2e40195430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2e30005610 tx=0x7f2e30005640 comp rx=0 tx=0).stop 2024-06-06T07:37:40.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 shutdown_connections 2024-06-06T07:37:40.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 --2- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2e2c037980 0x7f2e2c039e40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:40.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 --2- 172.21.15.184:0/2598073446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e400fff00 0x7f2e40195430 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:40.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 >> 172.21.15.184:0/2598073446 conn(0x7f2e400fb4a0 msgr2=0x7f2e400fbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:40.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 shutdown_connections 2024-06-06T07:37:40.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:40.423+0000 7f2e461be640 1 -- 172.21.15.184:0/2598073446 wait complete. 2024-06-06T07:37:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:40 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:40 smithi082 ceph-mon[29354]: mgrmap e17: smithi082.kocfig(active, since 2s) 2024-06-06T07:37:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:40 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/2598073446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:41.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:41.998 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:42.199 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:42.252 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T07:37:43.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:42 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:43.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:42 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:43.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:42 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:43.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:42 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:43.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:42 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:37:44.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.534+0000 7ff63b464640 1 -- 172.21.15.184:0/3148212173 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 msgr2=0x7ff6340ff3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:44.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.534+0000 7ff63b464640 1 --2- 172.21.15.184:0/3148212173 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 0x7ff6340ff3c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff630009880 tx=0x7ff63002a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:44.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.534+0000 7ff63b464640 1 -- 172.21.15.184:0/3148212173 shutdown_connections 2024-06-06T07:37:44.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.534+0000 7ff63b464640 1 --2- 172.21.15.184:0/3148212173 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 0x7ff6340ff3c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:44.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.534+0000 7ff63b464640 1 -- 172.21.15.184:0/3148212173 >> 172.21.15.184:0/3148212173 conn(0x7ff6340fa870 msgr2=0x7ff6340fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:44.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.534+0000 7ff63b464640 1 -- 172.21.15.184:0/3148212173 shutdown_connections 2024-06-06T07:37:44.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.534+0000 7ff63b464640 1 -- 172.21.15.184:0/3148212173 wait complete. 2024-06-06T07:37:44.536 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.535+0000 7ff63b464640 1 Processor -- start 2024-06-06T07:37:44.536 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.535+0000 7ff63b464640 1 -- start start 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.535+0000 7ff63b464640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 0x7ff634196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.535+0000 7ff63b464640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6340691f0 con 0x7ff6340fefe0 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.535+0000 7ff639200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 0x7ff634196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.535+0000 7ff639200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 0x7ff634196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:46266/0 (socket says 172.21.15.184:46266) 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.535+0000 7ff639200640 1 -- 172.21.15.184:0/1003307588 learned_addr learned my addr 172.21.15.184:0/1003307588 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.536+0000 7ff639200640 1 -- 172.21.15.184:0/1003307588 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630009560 con 0x7ff6340fefe0 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.536+0000 7ff639200640 1 --2- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 0x7ff634196690 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff630009230 tx=0x7ff6300056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.536+0000 7ff62a7fc640 1 -- 172.21.15.184:0/1003307588 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff630004aa0 con 0x7ff6340fefe0 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.536+0000 7ff62a7fc640 1 -- 172.21.15.184:0/1003307588 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff630004c00 con 0x7ff6340fefe0 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.536+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff634196c30 con 0x7ff6340fefe0 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.536+0000 7ff62a7fc640 1 -- 172.21.15.184:0/1003307588 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff630031940 con 0x7ff6340fefe0 2024-06-06T07:37:44.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.536+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6341970b0 con 0x7ff6340fefe0 2024-06-06T07:37:44.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.537+0000 7ff62a7fc640 1 -- 172.21.15.184:0/1003307588 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7ff630031b00 con 0x7ff6340fefe0 2024-06-06T07:37:44.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.537+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6340fb400 con 0x7ff6340fefe0 2024-06-06T07:37:44.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.537+0000 7ff62a7fc640 1 --2- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff620037980 0x7ff620039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:44.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.537+0000 7ff62a7fc640 1 -- 172.21.15.184:0/1003307588 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff63006b4f0 con 0x7ff6340fefe0 2024-06-06T07:37:44.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.537+0000 7ff6389ff640 1 --2- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff620037980 0x7ff620039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:44.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.538+0000 7ff6389ff640 1 --2- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff620037980 0x7ff620039e40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff6240098c0 tx=0x7ff624006db0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:44.542 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.542+0000 7ff62a7fc640 1 -- 172.21.15.184:0/1003307588 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff63003c470 con 0x7ff6340fefe0 2024-06-06T07:37:44.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:44.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:44.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:44.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:44.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:37:44.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:37:44.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: Updating smithi082:/etc/ceph/ceph.conf 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: Updating smithi184:/etc/ceph/ceph.conf 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi184", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi184", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-06-06T07:37:44.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:44 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:37:44.815 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.815+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6341004c0 con 0x7ff6340fefe0 2024-06-06T07:37:44.816 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.815+0000 7ff62a7fc640 1 -- 172.21.15.184:0/1003307588 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff630040300 con 0x7ff6340fefe0 2024-06-06T07:37:44.816 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:44.816 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:44.816 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:44.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.817+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff620037980 msgr2=0x7ff620039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:44.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.817+0000 7ff63b464640 1 --2- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff620037980 0x7ff620039e40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff6240098c0 tx=0x7ff624006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:44.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.817+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 msgr2=0x7ff634196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:44.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.817+0000 7ff63b464640 1 --2- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 0x7ff634196690 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff630009230 tx=0x7ff6300056e0 comp rx=0 tx=0).stop 2024-06-06T07:37:44.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.818+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 shutdown_connections 2024-06-06T07:37:44.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.818+0000 7ff63b464640 1 --2- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff620037980 0x7ff620039e40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:44.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.818+0000 7ff63b464640 1 --2- 172.21.15.184:0/1003307588 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6340fefe0 0x7ff634196690 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:44.819 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.818+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 >> 172.21.15.184:0/1003307588 conn(0x7ff6340fa870 msgr2=0x7ff634106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:44.819 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.818+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 shutdown_connections 2024-06-06T07:37:44.819 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:44.818+0000 7ff63b464640 1 -- 172.21.15.184:0/1003307588 wait complete. 2024-06-06T07:37:45.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:45 smithi082 ceph-mon[29354]: Updating smithi082:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:37:45.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:45 smithi082 ceph-mon[29354]: Updating smithi184:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:37:45.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:45 smithi082 ceph-mon[29354]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-06-06T07:37:45.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:45 smithi082 ceph-mon[29354]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-06-06T07:37:45.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:45 smithi082 ceph-mon[29354]: Updating smithi082:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-06T07:37:45.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:45 smithi082 ceph-mon[29354]: Updating smithi184:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-06T07:37:45.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:45 smithi082 ceph-mon[29354]: Deploying daemon ceph-exporter.smithi184 on smithi184 2024-06-06T07:37:45.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:45 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/1003307588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:46.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:46.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:46.943 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:37:48.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.408+0000 7fb480f02640 1 -- 172.21.15.184:0/2991022299 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 msgr2=0x7fb47c0ff3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:48.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.408+0000 7fb480f02640 1 --2- 172.21.15.184:0/2991022299 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 0x7fb47c0ff3e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009880 tx=0x7fb46c02a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:48.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.408+0000 7fb480f02640 1 -- 172.21.15.184:0/2991022299 shutdown_connections 2024-06-06T07:37:48.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.408+0000 7fb480f02640 1 --2- 172.21.15.184:0/2991022299 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 0x7fb47c0ff3e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:48.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.408+0000 7fb480f02640 1 -- 172.21.15.184:0/2991022299 >> 172.21.15.184:0/2991022299 conn(0x7fb47c0fa8d0 msgr2=0x7fb47c0fccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:48.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.408+0000 7fb480f02640 1 -- 172.21.15.184:0/2991022299 shutdown_connections 2024-06-06T07:37:48.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.408+0000 7fb480f02640 1 -- 172.21.15.184:0/2991022299 wait complete. 2024-06-06T07:37:48.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.409+0000 7fb480f02640 1 Processor -- start 2024-06-06T07:37:48.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.409+0000 7fb480f02640 1 -- start start 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.409+0000 7fb480f02640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 0x7fb47c196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.409+0000 7fb480f02640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c069240 con 0x7fb47c0ff000 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.409+0000 7fb47a59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 0x7fb47c196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.409+0000 7fb47a59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 0x7fb47c196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:46274/0 (socket says 172.21.15.184:46274) 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.409+0000 7fb47a59c640 1 -- 172.21.15.184:0/1420792464 learned_addr learned my addr 172.21.15.184:0/1420792464 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.410+0000 7fb47a59c640 1 -- 172.21.15.184:0/1420792464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c009560 con 0x7fb47c0ff000 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.410+0000 7fb47a59c640 1 --2- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 0x7fb47c196660 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009230 tx=0x7fb46c0056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.410+0000 7fb4737fe640 1 -- 172.21.15.184:0/1420792464 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb46c004aa0 con 0x7fb47c0ff000 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.410+0000 7fb4737fe640 1 -- 172.21.15.184:0/1420792464 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb46c004c00 con 0x7fb47c0ff000 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.410+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c196c00 con 0x7fb47c0ff000 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.410+0000 7fb4737fe640 1 -- 172.21.15.184:0/1420792464 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb46c031940 con 0x7fb47c0ff000 2024-06-06T07:37:48.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.410+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c197080 con 0x7fb47c0ff000 2024-06-06T07:37:48.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.411+0000 7fb4737fe640 1 -- 172.21.15.184:0/1420792464 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7fb46c031ae0 con 0x7fb47c0ff000 2024-06-06T07:37:48.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.411+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c1004e0 con 0x7fb47c0ff000 2024-06-06T07:37:48.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.411+0000 7fb4737fe640 1 --2- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb4680379d0 0x7fb468039e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:48.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.411+0000 7fb4737fe640 1 -- 172.21.15.184:0/1420792464 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fb46c06b640 con 0x7fb47c0ff000 2024-06-06T07:37:48.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.412+0000 7fb479d9b640 1 --2- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb4680379d0 0x7fb468039e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:48.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.412+0000 7fb479d9b640 1 --2- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb4680379d0 0x7fb468039e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb4640098c0 tx=0x7fb464006db0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:48.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.416+0000 7fb4737fe640 1 -- 172.21.15.184:0/1420792464 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb46c04d350 con 0x7fb47c0ff000 2024-06-06T07:37:48.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.672+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb47c061b30 con 0x7fb47c0ff000 2024-06-06T07:37:48.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.673+0000 7fb4737fe640 1 -- 172.21.15.184:0/1420792464 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb46c035030 con 0x7fb47c0ff000 2024-06-06T07:37:48.675 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:48.675 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:48.675 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:48.677 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.675+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb4680379d0 msgr2=0x7fb468039e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:48.677 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.675+0000 7fb480f02640 1 --2- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb4680379d0 0x7fb468039e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb4640098c0 tx=0x7fb464006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:48.677 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.675+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 msgr2=0x7fb47c196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:48.677 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.675+0000 7fb480f02640 1 --2- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 0x7fb47c196660 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009230 tx=0x7fb46c0056e0 comp rx=0 tx=0).stop 2024-06-06T07:37:48.677 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.675+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 shutdown_connections 2024-06-06T07:37:48.677 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.675+0000 7fb480f02640 1 --2- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb4680379d0 0x7fb468039e90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:48.677 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.675+0000 7fb480f02640 1 --2- 172.21.15.184:0/1420792464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb47c0ff000 0x7fb47c196660 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:48.677 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.675+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 >> 172.21.15.184:0/1420792464 conn(0x7fb47c0fa8d0 msgr2=0x7fb47c0fb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:48.678 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.676+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 shutdown_connections 2024-06-06T07:37:48.678 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:48.676+0000 7fb480f02640 1 -- 172.21.15.184:0/1420792464 wait complete. 2024-06-06T07:37:49.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:48 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/1420792464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:50.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:50.256 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:50.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:50.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:50.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:50.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:50.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi184", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T07:37:50.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi184", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-06T07:37:50.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:37:50.769 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:50 smithi082 ceph-mon[29354]: Deploying daemon crash.smithi184 on smithi184 2024-06-06T07:37:50.799 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:37:52.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.193+0000 7f9bfe045640 1 -- 172.21.15.184:0/790808666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 msgr2=0x7f9bf81051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:52.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.193+0000 7f9bfe045640 1 --2- 172.21.15.184:0/790808666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 0x7f9bf81051e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9be8009880 tx=0x7f9be802a5d0 comp rx=0 tx=0).stop 2024-06-06T07:37:52.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.193+0000 7f9bfe045640 1 -- 172.21.15.184:0/790808666 shutdown_connections 2024-06-06T07:37:52.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.193+0000 7f9bfe045640 1 --2- 172.21.15.184:0/790808666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 0x7f9bf81051e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:52.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.193+0000 7f9bfe045640 1 -- 172.21.15.184:0/790808666 >> 172.21.15.184:0/790808666 conn(0x7f9bf80748c0 msgr2=0x7f9bf8074cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:52.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.193+0000 7f9bfe045640 1 -- 172.21.15.184:0/790808666 shutdown_connections 2024-06-06T07:37:52.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.193+0000 7f9bfe045640 1 -- 172.21.15.184:0/790808666 wait complete. 2024-06-06T07:37:52.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.194+0000 7f9bfe045640 1 Processor -- start 2024-06-06T07:37:52.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.194+0000 7f9bfe045640 1 -- start start 2024-06-06T07:37:52.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.194+0000 7f9bfe045640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 0x7f9bf819b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:52.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.194+0000 7f9bfe045640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf819baa0 con 0x7f9bf8104e00 2024-06-06T07:37:52.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.194+0000 7f9bf77fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 0x7f9bf819b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:52.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.195+0000 7f9bf77fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 0x7f9bf819b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:46310/0 (socket says 172.21.15.184:46310) 2024-06-06T07:37:52.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.195+0000 7f9bf77fe640 1 -- 172.21.15.184:0/3808896392 learned_addr learned my addr 172.21.15.184:0/3808896392 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:52.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.195+0000 7f9bf77fe640 1 -- 172.21.15.184:0/3808896392 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be8009560 con 0x7f9bf8104e00 2024-06-06T07:37:52.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.195+0000 7f9bf77fe640 1 --2- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 0x7f9bf819b560 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9be8009230 tx=0x7f9be80056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:52.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.195+0000 7f9bf4ff9640 1 -- 172.21.15.184:0/3808896392 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9be8004aa0 con 0x7f9bf8104e00 2024-06-06T07:37:52.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.195+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf819bd00 con 0x7f9bf8104e00 2024-06-06T07:37:52.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.195+0000 7f9bf4ff9640 1 -- 172.21.15.184:0/3808896392 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9be8004c00 con 0x7f9bf8104e00 2024-06-06T07:37:52.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.196+0000 7f9bf4ff9640 1 -- 172.21.15.184:0/3808896392 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9be8031940 con 0x7f9bf8104e00 2024-06-06T07:37:52.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.196+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf8195610 con 0x7f9bf8104e00 2024-06-06T07:37:52.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.196+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf81004b0 con 0x7f9bf8104e00 2024-06-06T07:37:52.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.196+0000 7f9bf4ff9640 1 -- 172.21.15.184:0/3808896392 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7f9be80035f0 con 0x7f9bf8104e00 2024-06-06T07:37:52.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.197+0000 7f9bf4ff9640 1 --2- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9be0037980 0x7f9be0039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:52.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.197+0000 7f9bf4ff9640 1 -- 172.21.15.184:0/3808896392 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f9be806b4a0 con 0x7f9bf8104e00 2024-06-06T07:37:52.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.197+0000 7f9bf6ffd640 1 --2- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9be0037980 0x7f9be0039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:52.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.197+0000 7f9bf6ffd640 1 --2- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9be0037980 0x7f9be0039e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0098c0 tx=0x7f9bec006db0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:52.203 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.202+0000 7f9bf4ff9640 1 -- 172.21.15.184:0/3808896392 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9be803a7d0 con 0x7f9bf8104e00 2024-06-06T07:37:52.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.443+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bf8196c30 con 0x7f9bf8104e00 2024-06-06T07:37:52.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.465+0000 7f9bf4ff9640 1 -- 172.21.15.184:0/3808896392 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9bf8196c30 con 0x7f9bf8104e00 2024-06-06T07:37:52.695 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:52.695 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:52.696 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:52.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9be0037980 msgr2=0x7f9be0039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:52.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 --2- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9be0037980 0x7f9be0039e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0098c0 tx=0x7f9bec006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:52.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 msgr2=0x7f9bf819b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:52.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 --2- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 0x7f9bf819b560 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9be8009230 tx=0x7f9be80056e0 comp rx=0 tx=0).stop 2024-06-06T07:37:52.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 shutdown_connections 2024-06-06T07:37:52.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 --2- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9be0037980 0x7f9be0039e40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 --2- 172.21.15.184:0/3808896392 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8104e00 0x7f9bf819b560 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 >> 172.21.15.184:0/3808896392 conn(0x7f9bf80748c0 msgr2=0x7f9bf818e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 shutdown_connections 2024-06-06T07:37:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:52.467+0000 7f9bfe045640 1 -- 172.21.15.184:0/3808896392 wait complete. 2024-06-06T07:37:52.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:52 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/3808896392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:54.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:54.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:37:54.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:54.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:54.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:54.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:54.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:53 smithi082 ceph-mon[29354]: Deploying daemon node-exporter.smithi184 on smithi184 2024-06-06T07:37:54.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:54.143 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:54.342 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:37:56.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.771+0000 7fa19992e640 1 -- 172.21.15.184:0/2700304193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 msgr2=0x7fa194101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:56.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.771+0000 7fa19992e640 1 --2- 172.21.15.184:0/2700304193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 0x7fa194101490 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa184009880 tx=0x7fa18402a670 comp rx=0 tx=0).stop 2024-06-06T07:37:56.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.771+0000 7fa19992e640 1 -- 172.21.15.184:0/2700304193 shutdown_connections 2024-06-06T07:37:56.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.771+0000 7fa19992e640 1 --2- 172.21.15.184:0/2700304193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 0x7fa194101490 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:56.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.771+0000 7fa19992e640 1 -- 172.21.15.184:0/2700304193 >> 172.21.15.184:0/2700304193 conn(0x7fa1940fa8a0 msgr2=0x7fa1940fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:56.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.771+0000 7fa19992e640 1 -- 172.21.15.184:0/2700304193 shutdown_connections 2024-06-06T07:37:56.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.771+0000 7fa19992e640 1 -- 172.21.15.184:0/2700304193 wait complete. 2024-06-06T07:37:56.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.771+0000 7fa19992e640 1 Processor -- start 2024-06-06T07:37:56.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.771+0000 7fa19992e640 1 -- start start 2024-06-06T07:37:56.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.772+0000 7fa19992e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 0x7fa19419b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.772+0000 7fa19992e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19419ba40 con 0x7fa1941010b0 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.772+0000 7fa192ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 0x7fa19419b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.772+0000 7fa192ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 0x7fa19419b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:43252/0 (socket says 172.21.15.184:43252) 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.772+0000 7fa192ffd640 1 -- 172.21.15.184:0/183821295 learned_addr learned my addr 172.21.15.184:0/183821295 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.772+0000 7fa192ffd640 1 -- 172.21.15.184:0/183821295 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa184009560 con 0x7fa1941010b0 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.772+0000 7fa192ffd640 1 --2- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 0x7fa19419b500 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa184009230 tx=0x7fa1840056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.773+0000 7fa19892c640 1 -- 172.21.15.184:0/183821295 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa184004aa0 con 0x7fa1941010b0 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.773+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19419bc40 con 0x7fa1941010b0 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.773+0000 7fa19892c640 1 -- 172.21.15.184:0/183821295 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa184004c00 con 0x7fa1941010b0 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.773+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19419c060 con 0x7fa1941010b0 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.773+0000 7fa19892c640 1 -- 172.21.15.184:0/183821295 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa184031810 con 0x7fa1941010b0 2024-06-06T07:37:56.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.773+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa194068b30 con 0x7fa1941010b0 2024-06-06T07:37:56.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.774+0000 7fa19892c640 1 -- 172.21.15.184:0/183821295 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7fa184031990 con 0x7fa1941010b0 2024-06-06T07:37:56.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.774+0000 7fa19892c640 1 --2- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa180037980 0x7fa180039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:37:56.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.774+0000 7fa19892c640 1 -- 172.21.15.184:0/183821295 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa18406b710 con 0x7fa1941010b0 2024-06-06T07:37:56.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.774+0000 7fa18bfff640 1 --2- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa180037980 0x7fa180039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:37:56.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.774+0000 7fa18bfff640 1 --2- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa180037980 0x7fa180039e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0098c0 tx=0x7fa17c006db0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:37:56.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:56.779+0000 7fa19892c640 1 -- 172.21.15.184:0/183821295 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa1840439b0 con 0x7fa1941010b0 2024-06-06T07:37:57.021 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.021+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa194061ae0 con 0x7fa1941010b0 2024-06-06T07:37:57.022 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.021+0000 7fa19892c640 1 -- 172.21.15.184:0/183821295 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa184035070 con 0x7fa1941010b0 2024-06-06T07:37:57.022 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:37:57.022 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:37:57.022 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa180037980 msgr2=0x7fa180039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 --2- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa180037980 0x7fa180039e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0098c0 tx=0x7fa17c006db0 comp rx=0 tx=0).stop 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 msgr2=0x7fa19419b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 --2- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 0x7fa19419b500 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa184009230 tx=0x7fa1840056e0 comp rx=0 tx=0).stop 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 shutdown_connections 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 --2- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa180037980 0x7fa180039e40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 --2- 172.21.15.184:0/183821295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1941010b0 0x7fa19419b500 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 >> 172.21.15.184:0/183821295 conn(0x7fa1940fa8a0 msgr2=0x7fa194101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 shutdown_connections 2024-06-06T07:37:57.024 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:37:57.023+0000 7fa19992e640 1 -- 172.21.15.184:0/183821295 wait complete. 2024-06-06T07:37:57.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:57 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/183821295' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:37:58.617 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:37:58.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:37:59.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:37:59.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:59.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:59.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:59.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:59.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:37:59.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.cqttux", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T07:37:59.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.cqttux", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-06T07:37:59.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T07:37:59.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:37:59.146 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:38:00.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:37:59 smithi082 ceph-mon[29354]: Deploying daemon mgr.smithi184.cqttux on smithi184 2024-06-06T07:38:00.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.829+0000 7f466fda7640 1 -- 172.21.15.184:0/4080139135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 msgr2=0x7f46680ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:00.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.829+0000 7f466fda7640 1 --2- 172.21.15.184:0/4080139135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 0x7f46680ff3b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4664009c20 tx=0x7f466402aa90 comp rx=0 tx=0).stop 2024-06-06T07:38:00.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.829+0000 7f466fda7640 1 -- 172.21.15.184:0/4080139135 shutdown_connections 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.829+0000 7f466fda7640 1 --2- 172.21.15.184:0/4080139135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 0x7f46680ff3b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.829+0000 7f466fda7640 1 -- 172.21.15.184:0/4080139135 >> 172.21.15.184:0/4080139135 conn(0x7f46680fa8a0 msgr2=0x7f46680fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.829+0000 7f466fda7640 1 -- 172.21.15.184:0/4080139135 shutdown_connections 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.829+0000 7f466fda7640 1 -- 172.21.15.184:0/4080139135 wait complete. 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.830+0000 7f466fda7640 1 Processor -- start 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.830+0000 7f466fda7640 1 -- start start 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.830+0000 7f466fda7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 0x7f4668195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.830+0000 7f466fda7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4668195e70 con 0x7f46680fefd0 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.830+0000 7f466db43640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 0x7f4668195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.830+0000 7f466db43640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 0x7f4668195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:43272/0 (socket says 172.21.15.184:43272) 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.830+0000 7f466db43640 1 -- 172.21.15.184:0/935064220 learned_addr learned my addr 172.21.15.184:0/935064220 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:38:00.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.831+0000 7f466db43640 1 -- 172.21.15.184:0/935064220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664009900 con 0x7f46680fefd0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.831+0000 7f466db43640 1 --2- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 0x7f4668195930 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4664006010 tx=0x7f466402f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.831+0000 7f465effd640 1 -- 172.21.15.184:0/935064220 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466403e070 con 0x7f46680fefd0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.831+0000 7f465effd640 1 -- 172.21.15.184:0/935064220 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4664003e20 con 0x7f46680fefd0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.831+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4668196070 con 0x7f46680fefd0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.831+0000 7f465effd640 1 -- 172.21.15.184:0/935064220 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4664004670 con 0x7f46680fefd0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.831+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4668196490 con 0x7f46680fefd0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.832+0000 7f465effd640 1 -- 172.21.15.184:0/935064220 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7f466400a040 con 0x7f46680fefd0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.832+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46681004b0 con 0x7f46680fefd0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.832+0000 7f465effd640 1 --2- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4654037980 0x7f4654039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.832+0000 7f465effd640 1 -- 172.21.15.184:0/935064220 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f466406b8a0 con 0x7f46680fefd0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.832+0000 7f466d342640 1 --2- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4654037980 0x7f4654039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.833+0000 7f466d342640 1 --2- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4654037980 0x7f4654039e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f46580098c0 tx=0x7f4658006db0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:00.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:00.837+0000 7f465effd640 1 -- 172.21.15.184:0/935064220 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f466402fbf0 con 0x7f46680fefd0 2024-06-06T07:38:01.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:00 smithi082 ceph-mon[29354]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:01.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.115+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4668061ae0 con 0x7f46680fefd0 2024-06-06T07:38:01.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.116+0000 7f465effd640 1 -- 172.21.15.184:0/935064220 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f4664033030 con 0x7f46680fefd0 2024-06-06T07:38:01.117 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:38:01.117 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:38:01.117 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:38:01.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4654037980 msgr2=0x7f4654039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:01.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 --2- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4654037980 0x7f4654039e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f46580098c0 tx=0x7f4658006db0 comp rx=0 tx=0).stop 2024-06-06T07:38:01.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 msgr2=0x7f4668195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:01.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 --2- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 0x7f4668195930 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4664006010 tx=0x7f466402f8a0 comp rx=0 tx=0).stop 2024-06-06T07:38:01.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 shutdown_connections 2024-06-06T07:38:01.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 --2- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4654037980 0x7f4654039e40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:01.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 --2- 172.21.15.184:0/935064220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46680fefd0 0x7f4668195930 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:01.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 >> 172.21.15.184:0/935064220 conn(0x7f46680fa8a0 msgr2=0x7f46680682b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:01.120 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 shutdown_connections 2024-06-06T07:38:01.120 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:01.118+0000 7f466fda7640 1 -- 172.21.15.184:0/935064220 wait complete. 2024-06-06T07:38:02.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:01 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/935064220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:38:02.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:02.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:02.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:02.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:02.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T07:38:02.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:02.885 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:38:02.885 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:38:03.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:02 smithi082 ceph-mon[29354]: Deploying daemon mon.smithi184 on smithi184 2024-06-06T07:38:03.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:02 smithi082 ceph-mon[29354]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:03.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:03.540 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:38:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:04 smithi082 ceph-mon[29354]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:05.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.535+0000 7f169d16d640 1 -- 172.21.15.184:0/888564873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 msgr2=0x7f16981051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:05.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.535+0000 7f169d16d640 1 --2- 172.21.15.184:0/888564873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 0x7f16981051e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1688009880 tx=0x7f168802a5d0 comp rx=0 tx=0).stop 2024-06-06T07:38:05.537 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.535+0000 7f169d16d640 1 -- 172.21.15.184:0/888564873 shutdown_connections 2024-06-06T07:38:05.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.535+0000 7f169d16d640 1 --2- 172.21.15.184:0/888564873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 0x7f16981051e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:05.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.535+0000 7f169d16d640 1 -- 172.21.15.184:0/888564873 >> 172.21.15.184:0/888564873 conn(0x7f16980748c0 msgr2=0x7f1698074cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:05.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.535+0000 7f169d16d640 1 -- 172.21.15.184:0/888564873 shutdown_connections 2024-06-06T07:38:05.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.535+0000 7f169d16d640 1 -- 172.21.15.184:0/888564873 wait complete. 2024-06-06T07:38:05.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.536+0000 7f169d16d640 1 Processor -- start 2024-06-06T07:38:05.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.536+0000 7f169d16d640 1 -- start start 2024-06-06T07:38:05.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.536+0000 7f169d16d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 0x7f16981966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:05.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.536+0000 7f169d16d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1698068620 con 0x7f1698104e00 2024-06-06T07:38:05.538 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f1696d9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 0x7f16981966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:05.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f1696d9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 0x7f16981966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.184:56798/0 (socket says 172.21.15.184:56798) 2024-06-06T07:38:05.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f1696d9d640 1 -- 172.21.15.184:0/3384174965 learned_addr learned my addr 172.21.15.184:0/3384174965 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:38:05.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f1696d9d640 1 -- 172.21.15.184:0/3384174965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1688009560 con 0x7f1698104e00 2024-06-06T07:38:05.539 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f1696d9d640 1 --2- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 0x7f16981966d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1688009230 tx=0x7f16880056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f167ffff640 1 -- 172.21.15.184:0/3384174965 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1688004aa0 con 0x7f1698104e00 2024-06-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f167ffff640 1 -- 172.21.15.184:0/3384174965 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1688004c00 con 0x7f1698104e00 2024-06-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1698196c70 con 0x7f1698104e00 2024-06-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f167ffff640 1 -- 172.21.15.184:0/3384174965 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1688031940 con 0x7f1698104e00 2024-06-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.537+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16981970f0 con 0x7f1698104e00 2024-06-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.538+0000 7f167ffff640 1 -- 172.21.15.184:0/3384174965 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7f1688031b00 con 0x7f1698104e00 2024-06-06T07:38:05.540 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.538+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16981004b0 con 0x7f1698104e00 2024-06-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.539+0000 7f167ffff640 1 --2- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f1680037980 0x7f1680039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.539+0000 7f167ffff640 1 -- 172.21.15.184:0/3384174965 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f168806b510 con 0x7f1698104e00 2024-06-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.539+0000 7f169659c640 1 --2- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f1680037980 0x7f1680039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:05.541 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.539+0000 7f169659c640 1 --2- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f1680037980 0x7f1680039e40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f168c0098c0 tx=0x7f168c006db0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:05.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.543+0000 7f167ffff640 1 -- 172.21.15.184:0/3384174965 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f168803adc0 con 0x7f1698104e00 2024-06-06T07:38:05.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.801+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16981973a0 con 0x7f1698104e00 2024-06-06T07:38:05.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.802+0000 7f167ffff640 1 -- 172.21.15.184:0/3384174965 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1688035090 con 0x7f1698104e00 2024-06-06T07:38:05.803 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:38:05.803 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:35:25.881156Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T07:38:05.803 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f1680037980 msgr2=0x7f1680039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 --2- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f1680037980 0x7f1680039e40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f168c0098c0 tx=0x7f168c006db0 comp rx=0 tx=0).stop 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 msgr2=0x7f16981966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 --2- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 0x7f16981966d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1688009230 tx=0x7f16880056e0 comp rx=0 tx=0).stop 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 shutdown_connections 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 --2- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f1680037980 0x7f1680039e40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 --2- 172.21.15.184:0/3384174965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1698104e00 0x7f16981966d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 >> 172.21.15.184:0/3384174965 conn(0x7f16980748c0 msgr2=0x7f16980fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 shutdown_connections 2024-06-06T07:38:05.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:05.804+0000 7f169d16d640 1 -- 172.21.15.184:0/3384174965 wait complete. 2024-06-06T07:38:06.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:05 smithi184 ceph-mon[37288]: mon.smithi184@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-06-06T07:38:06.400 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:06 smithi184 bash[37235]: 551a4ac4a9ebac0b648e8affbfbd4bb9119c3b47c4732466c4626dea79710c9b 2024-06-06T07:38:06.400 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:06 smithi184 systemd[1]: Started Ceph mon.smithi184 for 355cdb60-23d7-11ef-bc9b-c7b262605968. 2024-06-06T07:38:07.574 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T07:38:07.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T07:38:07.776 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi184/config 2024-06-06T07:38:11.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:38:11.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: mon.smithi082 calling monitor election 2024-06-06T07:38:11.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: Failed to load ceph-mgr modules: k8sevents 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.? 172.21.15.184:0/4104892300' entity='mgr.smithi184.cqttux' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi184.cqttux/crt"}]: dispatch 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: mon.smithi184 calling monitor election 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: mon.smithi082 is new leader, mons smithi082,smithi184 in quorum (ranks 0,1) 2024-06-06T07:38:11.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: monmap e2: 2 mons at {smithi082=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0],smithi184=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: fsmap 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: osdmap e5: 0 total, 0 up, 0 in 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: mgrmap e17: smithi082.kocfig(active, since 33s) 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: mgr module k8sevents crashed in daemon mgr.smithi082.kocfig on host smithi082 at 2024-06-06T07:35:41.816792Z 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: mgr module k8sevents crashed in daemon mgr.smithi082.kocfig on host smithi082 at 2024-06-06T07:35:50.727922Z 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: mgr module k8sevents crashed in daemon mgr.smithi082.kocfig on host smithi082 at 2024-06-06T07:36:21.422796Z 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: Standby manager daemon smithi184.cqttux started 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.? 172.21.15.184:0/4104892300' entity='mgr.smithi184.cqttux' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.? 172.21.15.184:0/4104892300' entity='mgr.smithi184.cqttux' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi184.cqttux/key"}]: dispatch 2024-06-06T07:38:11.270 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.? 172.21.15.184:0/4104892300' entity='mgr.smithi184.cqttux' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-06T07:38:11.271 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:11.271 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:11.271 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:11.271 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:10 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:11.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:38:11.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: mon.smithi082 calling monitor election 2024-06-06T07:38:11.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: Failed to load ceph-mgr modules: k8sevents 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.? 172.21.15.184:0/4104892300' entity='mgr.smithi184.cqttux' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi184.cqttux/crt"}]: dispatch 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: mon.smithi184 calling monitor election 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: mon.smithi082 is new leader, mons smithi082,smithi184 in quorum (ranks 0,1) 2024-06-06T07:38:11.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: monmap e2: 2 mons at {smithi082=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0],smithi184=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: fsmap 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: osdmap e5: 0 total, 0 up, 0 in 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: mgrmap e17: smithi082.kocfig(active, since 33s) 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: mgr module k8sevents crashed in daemon mgr.smithi082.kocfig on host smithi082 at 2024-06-06T07:35:41.816792Z 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: mgr module k8sevents crashed in daemon mgr.smithi082.kocfig on host smithi082 at 2024-06-06T07:35:50.727922Z 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: mgr module k8sevents crashed in daemon mgr.smithi082.kocfig on host smithi082 at 2024-06-06T07:36:21.422796Z 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: Standby manager daemon smithi184.cqttux started 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.? 172.21.15.184:0/4104892300' entity='mgr.smithi184.cqttux' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.? 172.21.15.184:0/4104892300' entity='mgr.smithi184.cqttux' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi184.cqttux/key"}]: dispatch 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.? 172.21.15.184:0/4104892300' entity='mgr.smithi184.cqttux' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-06T07:38:11.346 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:11.347 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:11.347 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:11.347 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:10 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:12.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.168+0000 7f6e021ec640 1 -- 172.21.15.184:0/3880779683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6de8003530 msgr2=0x7f6de80059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:12.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.168+0000 7f6e021ec640 1 --2- 172.21.15.184:0/3880779683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6de8003530 0x7f6de80059c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0fedd0 tx=0x7f6dec02bce0 comp rx=0 tx=0).stop 2024-06-06T07:38:12.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.168+0000 7f6e021ec640 1 -- 172.21.15.184:0/3880779683 shutdown_connections 2024-06-06T07:38:12.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.168+0000 7f6e021ec640 1 --2- 172.21.15.184:0/3880779683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6de8003530 0x7f6de80059c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:12.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.168+0000 7f6e021ec640 1 -- 172.21.15.184:0/3880779683 >> 172.21.15.184:0/3880779683 conn(0x7f6dfc075b60 msgr2=0x7f6dfc0fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.168+0000 7f6e021ec640 1 -- 172.21.15.184:0/3880779683 shutdown_connections 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.169+0000 7f6e021ec640 1 -- 172.21.15.184:0/3880779683 wait complete. 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.169+0000 7f6e021ec640 1 Processor -- start 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.169+0000 7f6e021ec640 1 -- start start 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.169+0000 7f6e021ec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 0x7f6dfc195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.169+0000 7f6e021ec640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6dfc195fc0 0x7f6dfc19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.169+0000 7f6e021ec640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc1965e0 con 0x7f6dfc195fc0 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.169+0000 7f6e021ec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc196750 con 0x7f6de8003530 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.169+0000 7f6dfb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 0x7f6dfc195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6dfb7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 0x7f6dfc195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:60496/0 (socket says 172.21.15.184:60496) 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6dfb7fe640 1 -- 172.21.15.184:0/3594651502 learned_addr learned my addr 172.21.15.184:0/3594651502 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6dfb7fe640 1 -- 172.21.15.184:0/3594651502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 msgr2=0x7f6dfc195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-06-06T07:38:12.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6dfb7fe640 1 -- 172.21.15.184:0/3594651502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 msgr2=0x7f6dfc195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-06T07:38:12.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6dfb7fe640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 0x7f6dfc195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-06T07:38:12.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6dfb7fe640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 0x7f6dfc195a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T07:38:12.172 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6df3fff640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6dfc195fc0 0x7f6dfc19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:12.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6df3fff640 1 -- 172.21.15.184:0/3594651502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 msgr2=0x7f6dfc195a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T07:38:12.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6df3fff640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 0x7f6dfc195a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:12.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6df3fff640 1 -- 172.21.15.184:0/3594651502 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec011870 con 0x7f6dfc195fc0 2024-06-06T07:38:12.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6df3fff640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6dfc195fc0 0x7f6dfc19a2b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6de400b3d0 tx=0x7f6de400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:12.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.170+0000 7f6df97fa640 1 -- 172.21.15.184:0/3594651502 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6de4011800 con 0x7f6dfc195fc0 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.171+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc19a8b0 con 0x7f6dfc195fc0 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.171+0000 7f6df97fa640 1 -- 172.21.15.184:0/3594651502 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6de4011e20 con 0x7f6dfc195fc0 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.171+0000 7f6df97fa640 1 -- 172.21.15.184:0/3594651502 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6de400f2e0 con 0x7f6dfc195fc0 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.171+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc19ae00 con 0x7f6dfc195fc0 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.172+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc103790 con 0x7f6dfc195fc0 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.172+0000 7f6df97fa640 1 -- 172.21.15.184:0/3594651502 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7f6de4010380 con 0x7f6dfc195fc0 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.172+0000 7f6df97fa640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6ddc06afc0 0x7f6ddc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.172+0000 7f6df97fa640 1 -- 172.21.15.184:0/3594651502 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f6de4088760 con 0x7f6dfc195fc0 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.172+0000 7f6dfb7fe640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6ddc06afc0 0x7f6ddc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.173+0000 7f6dfb7fe640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6ddc06afc0 0x7f6ddc06d480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6dec005940 tx=0x7f6dec0058b0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:12.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.177+0000 7f6df97fa640 1 -- 172.21.15.184:0/3594651502 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6de4054ae0 con 0x7f6dfc195fc0 2024-06-06T07:38:12.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:12 smithi082 ceph-mon[29354]: mgrmap e18: smithi082.kocfig(active, since 33s), standbys: smithi184.cqttux 2024-06-06T07:38:12.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:12 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr metadata", "who": "smithi184.cqttux", "id": "smithi184.cqttux"}]: dispatch 2024-06-06T07:38:12.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:12 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:38:12.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:12 smithi082 ceph-mon[29354]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:12.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:12 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:12.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:12 smithi184 ceph-mon[37288]: mgrmap e18: smithi082.kocfig(active, since 33s), standbys: smithi184.cqttux 2024-06-06T07:38:12.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:12 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr metadata", "who": "smithi184.cqttux", "id": "smithi184.cqttux"}]: dispatch 2024-06-06T07:38:12.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:12 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:38:12.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:12 smithi184 ceph-mon[37288]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:12.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:12 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:38:12.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.421+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6dfc1005e0 con 0x7f6dfc195fc0 2024-06-06T07:38:12.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.422+0000 7f6df97fa640 1 -- 172.21.15.184:0/3594651502 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1022 (secure 0 0 0) 0x7f6de4087f20 con 0x7f6dfc195fc0 2024-06-06T07:38:12.423 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-06-06T07:38:12.423 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":2,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","modified":"2024-06-06T07:38:05.927649Z","created":"2024-06-06T07:35:25.881156Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi082","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi184","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-06T07:38:12.423 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 2 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6ddc06afc0 msgr2=0x7f6ddc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6ddc06afc0 0x7f6ddc06d480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6dec005940 tx=0x7f6dec0058b0 comp rx=0 tx=0).stop 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6dfc195fc0 msgr2=0x7f6dfc19a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6dfc195fc0 0x7f6dfc19a2b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6de400b3d0 tx=0x7f6de400b890 comp rx=0 tx=0).stop 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 shutdown_connections 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6ddc06afc0 0x7f6ddc06d480 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6dfc195fc0 0x7f6dfc19a2b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 --2- 172.21.15.184:0/3594651502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6de8003530 0x7f6dfc195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 >> 172.21.15.184:0/3594651502 conn(0x7f6dfc075b60 msgr2=0x7f6dfc068430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 shutdown_connections 2024-06-06T07:38:12.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:38:12.424+0000 7f6e021ec640 1 -- 172.21.15.184:0/3594651502 wait complete. 2024-06-06T07:38:13.202 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-06T07:38:13.202 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-06-06T07:38:13.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:13 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/3594651502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:38:13.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:13 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:13.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:13 smithi184 ceph-mon[37288]: from='client.? 172.21.15.184:0/3594651502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T07:38:13.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:13 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:13.427 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:38:14.469 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: Updating smithi082:/etc/ceph/ceph.conf 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: Updating smithi184:/etc/ceph/ceph.conf 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: Updating smithi184:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: Updating smithi082:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.470 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.471 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T07:38:14.471 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T07:38:14.471 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:14.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: Updating smithi082:/etc/ceph/ceph.conf 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: Updating smithi184:/etc/ceph/ceph.conf 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: Updating smithi184:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: Updating smithi082:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:14.596 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T07:38:14.596 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T07:38:14.596 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:15.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:15 smithi082 ceph-mon[29354]: Reconfiguring mon.smithi082 (unknown last config time)... 2024-06-06T07:38:15.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:15 smithi082 ceph-mon[29354]: Reconfiguring daemon mon.smithi082 on smithi082 2024-06-06T07:38:15.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:15 smithi184 ceph-mon[37288]: Reconfiguring mon.smithi082 (unknown last config time)... 2024-06-06T07:38:15.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:15 smithi184 ceph-mon[37288]: Reconfiguring daemon mon.smithi082 on smithi082 2024-06-06T07:38:16.079 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.070+0000 7fb45550d640 1 -- 172.21.15.82:0/482537668 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 msgr2=0x7fb4500ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:16.079 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.070+0000 7fb45550d640 1 --2- 172.21.15.82:0/482537668 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 0x7fb4500ff780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb440009880 tx=0x7fb44002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:38:16.079 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.070+0000 7fb45550d640 1 -- 172.21.15.82:0/482537668 shutdown_connections 2024-06-06T07:38:16.079 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.070+0000 7fb45550d640 1 --2- 172.21.15.82:0/482537668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4500ffd50 0x7fb450103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:16.079 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.070+0000 7fb45550d640 1 --2- 172.21.15.82:0/482537668 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 0x7fb4500ff780 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:16.079 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.070+0000 7fb45550d640 1 -- 172.21.15.82:0/482537668 >> 172.21.15.82:0/482537668 conn(0x7fb4500fac70 msgr2=0x7fb4500fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.070+0000 7fb45550d640 1 -- 172.21.15.82:0/482537668 shutdown_connections 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.070+0000 7fb45550d640 1 -- 172.21.15.82:0/482537668 wait complete. 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.070+0000 7fb45550d640 1 Processor -- start 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb45550d640 1 -- start start 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb45550d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 0x7fb450100e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb45550d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4500ffd50 0x7fb450101380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb45550d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb450102e00 con 0x7fb4500ff3a0 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb45550d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4501018c0 con 0x7fb4500ffd50 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb44effd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 0x7fb450100e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb44effd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 0x7fb450100e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:41762/0 (socket says 172.21.15.82:41762) 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb44effd640 1 -- 172.21.15.82:0/3854525388 learned_addr learned my addr 172.21.15.82:0/3854525388 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:38:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb44e7fc640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4500ffd50 0x7fb450101380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb44effd640 1 -- 172.21.15.82:0/3854525388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4500ffd50 msgr2=0x7fb450101380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb44effd640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4500ffd50 0x7fb450101380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb44effd640 1 -- 172.21.15.82:0/3854525388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb440009560 con 0x7fb4500ff3a0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb44e7fc640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4500ffd50 0x7fb450101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb44effd640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 0x7fb450100e40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb44002a9c0 tx=0x7fb4400055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb437fff640 1 -- 172.21.15.82:0/3854525388 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb440004aa0 con 0x7fb4500ff3a0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb450101b40 con 0x7fb4500ff3a0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb437fff640 1 -- 172.21.15.82:0/3854525388 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb440004c00 con 0x7fb4500ff3a0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.071+0000 7fb437fff640 1 -- 172.21.15.82:0/3854525388 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb440003ab0 con 0x7fb4500ff3a0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.072+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4501a2690 con 0x7fb4500ff3a0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.072+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb450100880 con 0x7fb4500ff3a0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.073+0000 7fb437fff640 1 -- 172.21.15.82:0/3854525388 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7fb440003ca0 con 0x7fb4500ff3a0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.073+0000 7fb437fff640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb43806ae70 0x7fb43806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.073+0000 7fb437fff640 1 -- 172.21.15.82:0/3854525388 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fb4400a9f20 con 0x7fb4500ff3a0 2024-06-06T07:38:16.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.073+0000 7fb44e7fc640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb43806ae70 0x7fb43806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:16.082 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.074+0000 7fb44e7fc640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb43806ae70 0x7fb43806d330 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb450102520 tx=0x7fb444005d30 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:16.082 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.078+0000 7fb437fff640 1 -- 172.21.15.82:0/3854525388 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb4400762a0 con 0x7fb4500ff3a0 2024-06-06T07:38:16.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.255+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb45004e210 con 0x7fb4500ff3a0 2024-06-06T07:38:16.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.256+0000 7fb437fff640 1 -- 172.21.15.82:0/3854525388 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7fb4400a96e0 con 0x7fb4500ff3a0 2024-06-06T07:38:16.256 INFO:teuthology.orchestra.run.smithi082.stdout:# minimal ceph.conf for 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:38:16.256 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2024-06-06T07:38:16.256 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:38:16.257 INFO:teuthology.orchestra.run.smithi082.stdout: mon_host = [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2024-06-06T07:38:16.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb43806ae70 msgr2=0x7fb43806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:16.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb43806ae70 0x7fb43806d330 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb450102520 tx=0x7fb444005d30 comp rx=0 tx=0).stop 2024-06-06T07:38:16.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 msgr2=0x7fb450100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:16.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 0x7fb450100e40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb44002a9c0 tx=0x7fb4400055a0 comp rx=0 tx=0).stop 2024-06-06T07:38:16.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 shutdown_connections 2024-06-06T07:38:16.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb43806ae70 0x7fb43806d330 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:16.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4500ffd50 0x7fb450101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:16.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 --2- 172.21.15.82:0/3854525388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4500ff3a0 0x7fb450100e40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:16.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 >> 172.21.15.82:0/3854525388 conn(0x7fb4500fac70 msgr2=0x7fb450068e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:16.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 shutdown_connections 2024-06-06T07:38:16.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:16.259+0000 7fb45550d640 1 -- 172.21.15.82:0/3854525388 wait complete. 2024-06-06T07:38:16.717 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:16 smithi082 ceph-mon[29354]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:16.717 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:16 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:16.717 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:16 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:16.717 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:16 smithi082 ceph-mon[29354]: Reconfiguring mgr.smithi082.kocfig (unknown last config time)... 2024-06-06T07:38:16.717 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:16 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi082.kocfig", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T07:38:16.717 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:16 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T07:38:16.717 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:16 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:16.718 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:16 smithi082 ceph-mon[29354]: Reconfiguring daemon mgr.smithi082.kocfig on smithi082 2024-06-06T07:38:16.718 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:16 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3854525388' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:17.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:16 smithi184 ceph-mon[37288]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:17.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:16 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:17.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:16 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:17.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:16 smithi184 ceph-mon[37288]: Reconfiguring mgr.smithi082.kocfig (unknown last config time)... 2024-06-06T07:38:17.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:16 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi082.kocfig", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T07:38:17.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:16 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T07:38:17.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:16 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:17.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:16 smithi184 ceph-mon[37288]: Reconfiguring daemon mgr.smithi082.kocfig on smithi082 2024-06-06T07:38:17.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:16 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3854525388' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:17.249 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-06T07:38:17.250 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:38:17.250 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.conf 2024-06-06T07:38:17.277 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:38:17.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-06T07:38:17.342 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:38:17.343 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.conf 2024-06-06T07:38:17.376 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:38:17.376 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-06T07:38:17.443 INFO:tasks.cephadm:Deploying OSDs... 2024-06-06T07:38:17.443 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:38:17.443 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-06-06T07:38:17.458 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-06T07:38:17.459 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-06-06T07:38:17.516 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-06T07:38:17.516 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T07:38:17.516 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-06T07:38:17.516 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T07:38:17.516 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T07:38:17.516 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-06-06 07:37:31.030924025 +0000 2024-06-06T07:38:17.516 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-06-06 07:31:49.388828036 +0000 2024-06-06T07:38:17.516 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-06-06 07:31:49.388828036 +0000 2024-06-06T07:38:17.516 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-06-06 07:31:49.388828036 +0000 2024-06-06T07:38:17.517 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-06T07:38:17.580 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-06-06T07:38:17.580 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-06-06T07:38:17.580 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000190092 s, 2.7 MB/s 2024-06-06T07:38:17.581 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-06T07:38:17.637 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-06-06T07:38:17.693 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-06T07:38:17.693 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T07:38:17.693 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-06-06T07:38:17.693 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T07:38:17.693 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T07:38:17.693 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-06-06 07:37:31.030924025 +0000 2024-06-06T07:38:17.693 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-06-06 07:31:49.725821649 +0000 2024-06-06T07:38:17.694 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-06-06 07:31:49.725821649 +0000 2024-06-06T07:38:17.694 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-06-06 07:31:49.725821649 +0000 2024-06-06T07:38:17.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-06T07:38:17.756 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-06-06T07:38:17.756 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-06-06T07:38:17.757 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000168812 s, 3.0 MB/s 2024-06-06T07:38:17.758 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-06T07:38:17.814 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-06-06T07:38:17.869 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-06T07:38:17.869 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T07:38:17.869 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-06-06T07:38:17.869 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T07:38:17.869 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T07:38:17.869 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-06-06 07:37:31.031924007 +0000 2024-06-06T07:38:17.869 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-06-06 07:31:50.061815282 +0000 2024-06-06T07:38:17.870 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-06-06 07:31:50.061815282 +0000 2024-06-06T07:38:17.870 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-06-06 07:31:50.061815282 +0000 2024-06-06T07:38:17.870 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-06T07:38:17.935 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-06-06T07:38:17.935 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-06-06T07:38:17.935 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000165683 s, 3.1 MB/s 2024-06-06T07:38:17.937 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-06T07:38:17.993 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-06-06T07:38:18.049 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-06T07:38:18.049 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T07:38:18.049 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-06T07:38:18.049 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T07:38:18.049 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T07:38:18.049 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-06-06 07:37:31.031924007 +0000 2024-06-06T07:38:18.049 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-06-06 07:31:50.398808896 +0000 2024-06-06T07:38:18.049 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-06-06 07:31:50.398808896 +0000 2024-06-06T07:38:18.050 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-06-06 07:31:50.398808896 +0000 2024-06-06T07:38:18.050 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-06T07:38:18.112 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-06-06T07:38:18.113 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-06-06T07:38:18.113 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000190851 s, 2.7 MB/s 2024-06-06T07:38:18.114 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-06T07:38:18.170 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:38:18.170 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-06-06T07:38:18.189 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-06T07:38:18.189 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-06-06T07:38:18.249 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-06T07:38:18.249 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T07:38:18.249 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 676 Links: 1 2024-06-06T07:38:18.250 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T07:38:18.250 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T07:38:18.250 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-06-06 07:38:04.221675879 +0000 2024-06-06T07:38:18.250 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-06-06 07:31:49.365953365 +0000 2024-06-06T07:38:18.250 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-06-06 07:31:49.365953365 +0000 2024-06-06T07:38:18.250 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-06-06 07:31:49.365953365 +0000 2024-06-06T07:38:18.250 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-06T07:38:18.317 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-06-06T07:38:18.317 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-06-06T07:38:18.317 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000216653 s, 2.4 MB/s 2024-06-06T07:38:18.319 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-06T07:38:18.379 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-06-06T07:38:18.439 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-06T07:38:18.439 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T07:38:18.439 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 693 Links: 1 2024-06-06T07:38:18.439 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T07:38:18.439 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T07:38:18.439 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-06-06 07:38:04.222675859 +0000 2024-06-06T07:38:18.439 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-06-06 07:31:49.711946124 +0000 2024-06-06T07:38:18.439 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-06-06 07:31:49.711946124 +0000 2024-06-06T07:38:18.439 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-06-06 07:31:49.711946124 +0000 2024-06-06T07:38:18.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-06T07:38:18.507 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-06-06T07:38:18.507 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-06-06T07:38:18.507 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00023059 s, 2.2 MB/s 2024-06-06T07:38:18.509 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-06T07:38:18.569 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-06-06T07:38:18.635 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-06T07:38:18.635 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T07:38:18.635 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 711 Links: 1 2024-06-06T07:38:18.635 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T07:38:18.635 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T07:38:18.635 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-06-06 07:38:04.222675859 +0000 2024-06-06T07:38:18.635 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-06-06 07:31:50.040939239 +0000 2024-06-06T07:38:18.635 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-06-06 07:31:50.040939239 +0000 2024-06-06T07:38:18.635 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-06-06 07:31:50.040939239 +0000 2024-06-06T07:38:18.636 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-06T07:38:18.708 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-06-06T07:38:18.708 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-06-06T07:38:18.708 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00020222 s, 2.5 MB/s 2024-06-06T07:38:18.710 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-06T07:38:18.771 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-06-06T07:38:18.835 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-06T07:38:18.835 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T07:38:18.835 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 731 Links: 1 2024-06-06T07:38:18.835 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T07:38:18.835 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T07:38:18.835 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-06-06 07:38:04.222675859 +0000 2024-06-06T07:38:18.835 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-06-06 07:31:50.375932229 +0000 2024-06-06T07:38:18.835 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-06-06 07:31:50.375932229 +0000 2024-06-06T07:38:18.835 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-06-06 07:31:50.375932229 +0000 2024-06-06T07:38:18.836 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-06T07:38:18.908 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-06-06T07:38:18.908 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-06-06T07:38:18.908 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000215929 s, 2.4 MB/s 2024-06-06T07:38:18.910 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-06T07:38:18.971 INFO:tasks.cephadm:Deploying osd.0 on smithi082 with /dev/vg_nvme/lv_4... 2024-06-06T07:38:18.971 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-06T07:38:18.995 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:18 smithi082 ceph-mon[29354]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:18.995 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:18.995 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:18.995 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:18 smithi082 ceph-mon[29354]: Reconfiguring ceph-exporter.smithi082 (monmap changed)... 2024-06-06T07:38:18.995 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi082", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-06T07:38:18.995 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:18.995 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:18 smithi082 ceph-mon[29354]: Reconfiguring daemon ceph-exporter.smithi082 on smithi082 2024-06-06T07:38:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:18 smithi184 ceph-mon[37288]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:18 smithi184 ceph-mon[37288]: Reconfiguring ceph-exporter.smithi082 (monmap changed)... 2024-06-06T07:38:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi082", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-06T07:38:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:19.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:18 smithi184 ceph-mon[37288]: Reconfiguring daemon ceph-exporter.smithi082 on smithi082 2024-06-06T07:38:19.753 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:38:20.169 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:19 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:20.169 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:19 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:20.169 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:19 smithi082 ceph-mon[29354]: Reconfiguring crash.smithi082 (monmap changed)... 2024-06-06T07:38:20.170 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:19 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi082", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T07:38:20.170 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:19 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:20.170 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:19 smithi082 ceph-mon[29354]: Reconfiguring daemon crash.smithi082 on smithi082 2024-06-06T07:38:20.170 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:19 smithi082 ceph-mon[29354]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:20.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:19 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:20.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:19 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:20.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:19 smithi184 ceph-mon[37288]: Reconfiguring crash.smithi082 (monmap changed)... 2024-06-06T07:38:20.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:19 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi082", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T07:38:20.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:19 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:20.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:19 smithi184 ceph-mon[37288]: Reconfiguring daemon crash.smithi082 on smithi082 2024-06-06T07:38:20.345 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:19 smithi184 ceph-mon[37288]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:22.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:21 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:22.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:21 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:22.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:21 smithi082 ceph-mon[29354]: Reconfiguring alertmanager.smithi082 (dependencies changed)... 2024-06-06T07:38:22.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:21 smithi082 ceph-mon[29354]: Reconfiguring daemon alertmanager.smithi082 on smithi082 2024-06-06T07:38:22.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:21 smithi082 ceph-mon[29354]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:22.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:21 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:22.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:21 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:22.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:21 smithi184 ceph-mon[37288]: Reconfiguring alertmanager.smithi082 (dependencies changed)... 2024-06-06T07:38:22.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:21 smithi184 ceph-mon[37288]: Reconfiguring daemon alertmanager.smithi082 on smithi082 2024-06-06T07:38:22.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:21 smithi184 ceph-mon[37288]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:22.647 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-06T07:38:22.736 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-06T07:38:22.750 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-06-06T07:38:22.750 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-06-06T07:38:22.750 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123334 s, 850 MB/s 2024-06-06T07:38:22.751 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-06-06T07:38:23.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:38:23.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:38:24.054 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:23 smithi082 ceph-mon[29354]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:24.318 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_4 2024-06-06T07:38:24.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:23 smithi184 ceph-mon[37288]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:25.003 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:38:26.466 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 -- 172.21.15.82:0/3600927350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58100d10 msgr2=0x7fce58103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 --2- 172.21.15.82:0/3600927350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58100d10 0x7fce58103100 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce48009880 tx=0x7fce4802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 -- 172.21.15.82:0/3600927350 shutdown_connections 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 --2- 172.21.15.82:0/3600927350 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce58103640 0x7fce58105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 --2- 172.21.15.82:0/3600927350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58100d10 0x7fce58103100 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 -- 172.21.15.82:0/3600927350 >> 172.21.15.82:0/3600927350 conn(0x7fce580fa700 msgr2=0x7fce580fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 -- 172.21.15.82:0/3600927350 shutdown_connections 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 -- 172.21.15.82:0/3600927350 wait complete. 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 Processor -- start 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.463+0000 7fce5ea52640 1 -- start start 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce5ea52640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce58100d10 0x7fce58100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce5ea52640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58103640 0x7fce580fecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:26.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce5ea52640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580ff200 con 0x7fce58103640 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce5ea52640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580ff370 con 0x7fce58100d10 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce577fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58103640 0x7fce580fecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce577fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58103640 0x7fce580fecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:60606/0 (socket says 172.21.15.82:60606) 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce577fe640 1 -- 172.21.15.82:0/3023596910 learned_addr learned my addr 172.21.15.82:0/3023596910 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce57fff640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce58100d10 0x7fce58100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce577fe640 1 -- 172.21.15.82:0/3023596910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce58100d10 msgr2=0x7fce58100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce577fe640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce58100d10 0x7fce58100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce577fe640 1 -- 172.21.15.82:0/3023596910 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce48009560 con 0x7fce58103640 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce57fff640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce58100d10 0x7fce58100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.464+0000 7fce577fe640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58103640 0x7fce580fecc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00d6f0 tx=0x7fce4c00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:26.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.465+0000 7fce557fa640 1 -- 172.21.15.82:0/3023596910 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce4c009a30 con 0x7fce58103640 2024-06-06T07:38:26.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.465+0000 7fce557fa640 1 -- 172.21.15.82:0/3023596910 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce4c009b90 con 0x7fce58103640 2024-06-06T07:38:26.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.465+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce580ff650 con 0x7fce58103640 2024-06-06T07:38:26.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.465+0000 7fce557fa640 1 -- 172.21.15.82:0/3023596910 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce4c00f680 con 0x7fce58103640 2024-06-06T07:38:26.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.465+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5819de00 con 0x7fce58103640 2024-06-06T07:38:26.470 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.466+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce58108e70 con 0x7fce58103640 2024-06-06T07:38:26.470 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.466+0000 7fce557fa640 1 -- 172.21.15.82:0/3023596910 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7fce4c009d00 con 0x7fce58103640 2024-06-06T07:38:26.470 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.467+0000 7fce557fa640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fce4006ae60 0x7fce4006d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:38:26.470 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.467+0000 7fce557fa640 1 -- 172.21.15.82:0/3023596910 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fce4c088c60 con 0x7fce58103640 2024-06-06T07:38:26.470 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.467+0000 7fce57fff640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fce4006ae60 0x7fce4006d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:38:26.470 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.467+0000 7fce57fff640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fce4006ae60 0x7fce4006d320 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fce48006010 tx=0x7fce48000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:38:26.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.471+0000 7fce557fa640 1 -- 172.21.15.82:0/3023596910 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fce4c055060 con 0x7fce58103640 2024-06-06T07:38:26.664 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:26.662+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 --> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fce581040e0 con 0x7fce4006ae60 2024-06-06T07:38:26.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:26 smithi184 ceph-mon[37288]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:26.904 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:26 smithi082 ceph-mon[29354]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:27.733 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:27 smithi082 ceph-mon[29354]: from='client.14260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:38:27.733 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:27 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:38:27.733 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:27 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:38:27.733 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:27 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:27.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:27 smithi184 ceph-mon[37288]: from='client.14260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:38:27.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:27 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:38:27.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:27 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:38:27.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:27 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:28.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:28 smithi082 ceph-mon[29354]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:28.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:28 smithi184 ceph-mon[37288]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:30.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:30.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:30.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:30 smithi082 ceph-mon[29354]: Reconfiguring grafana.smithi082 (dependencies changed)... 2024-06-06T07:38:30.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:30 smithi082 ceph-mon[29354]: Reconfiguring daemon grafana.smithi082 on smithi082 2024-06-06T07:38:30.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:30 smithi082 ceph-mon[29354]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:30.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:30.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:30.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:30 smithi184 ceph-mon[37288]: Reconfiguring grafana.smithi082 (dependencies changed)... 2024-06-06T07:38:30.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:30 smithi184 ceph-mon[37288]: Reconfiguring daemon grafana.smithi082 on smithi082 2024-06-06T07:38:30.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:30 smithi184 ceph-mon[37288]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:31.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:31 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1728019056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "919724ed-ba2c-4df8-aa46-cd599dcf0647"}]: dispatch 2024-06-06T07:38:31.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:31 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1728019056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "919724ed-ba2c-4df8-aa46-cd599dcf0647"}]': finished 2024-06-06T07:38:31.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:31 smithi184 ceph-mon[37288]: osdmap e6: 1 total, 0 up, 1 in 2024-06-06T07:38:31.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:31 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:31.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:31 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1138240118' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:38:31.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:31 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1728019056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "919724ed-ba2c-4df8-aa46-cd599dcf0647"}]: dispatch 2024-06-06T07:38:31.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:31 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1728019056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "919724ed-ba2c-4df8-aa46-cd599dcf0647"}]': finished 2024-06-06T07:38:31.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:31 smithi082 ceph-mon[29354]: osdmap e6: 1 total, 0 up, 1 in 2024-06-06T07:38:31.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:31 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:31.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:31 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1138240118' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:38:32.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:32 smithi184 ceph-mon[37288]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:32.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:32 smithi082 ceph-mon[29354]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:34.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:34 smithi184 ceph-mon[37288]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:34.858 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:34 smithi082 ceph-mon[29354]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:36.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:36 smithi082 ceph-mon[29354]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:36.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:36 smithi184 ceph-mon[37288]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:37.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:38:37.948 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:38:38.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:38 smithi082 ceph-mon[29354]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:39.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:38 smithi184 ceph-mon[37288]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:40.730 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:40 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:40.731 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:40 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:40.731 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:40 smithi082 ceph-mon[29354]: Reconfiguring prometheus.smithi082 (dependencies changed)... 2024-06-06T07:38:40.731 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:40 smithi082 ceph-mon[29354]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:40.731 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:40 smithi082 ceph-mon[29354]: Reconfiguring daemon prometheus.smithi082 on smithi082 2024-06-06T07:38:40.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:40 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:40.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:40 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:40.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:40 smithi184 ceph-mon[37288]: Reconfiguring prometheus.smithi082 (dependencies changed)... 2024-06-06T07:38:40.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:40 smithi184 ceph-mon[37288]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:40.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:40 smithi184 ceph-mon[37288]: Reconfiguring daemon prometheus.smithi082 on smithi082 2024-06-06T07:38:41.610 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-06T07:38:41.610 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:41.846 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-06T07:38:41.847 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:42.746 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:42 smithi082 ceph-mon[29354]: Deploying daemon osd.0 on smithi082 2024-06-06T07:38:42.746 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:42 smithi082 ceph-mon[29354]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:42.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:42 smithi184 ceph-mon[37288]: Deploying daemon osd.0 on smithi082 2024-06-06T07:38:42.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:42 smithi184 ceph-mon[37288]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:44.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:44 smithi184 ceph-mon[37288]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:44.960 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:44 smithi082 ceph-mon[29354]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:46.460 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:46 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:46.460 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:46 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:46.461 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:46 smithi082 ceph-mon[29354]: Reconfiguring ceph-exporter.smithi184 (monmap changed)... 2024-06-06T07:38:46.461 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:46 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi184", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-06T07:38:46.461 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:46 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:46.461 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:46 smithi082 ceph-mon[29354]: Reconfiguring daemon ceph-exporter.smithi184 on smithi184 2024-06-06T07:38:46.461 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:46 smithi082 ceph-mon[29354]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:46.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:46 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:46.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:46 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:46.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:46 smithi184 ceph-mon[37288]: Reconfiguring ceph-exporter.smithi184 (monmap changed)... 2024-06-06T07:38:46.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:46 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi184", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-06T07:38:46.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:46 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:46.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:46 smithi184 ceph-mon[37288]: Reconfiguring daemon ceph-exporter.smithi184 on smithi184 2024-06-06T07:38:46.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:46 smithi184 ceph-mon[37288]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:47.709 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:47 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:47.709 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:47 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:47.709 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:47 smithi082 ceph-mon[29354]: Reconfiguring crash.smithi184 (monmap changed)... 2024-06-06T07:38:47.709 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:47 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi184", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T07:38:47.709 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:47 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:47.709 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:47 smithi082 ceph-mon[29354]: Reconfiguring daemon crash.smithi184 on smithi184 2024-06-06T07:38:47.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:47 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:47.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:47 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:47.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:47 smithi184 ceph-mon[37288]: Reconfiguring crash.smithi184 (monmap changed)... 2024-06-06T07:38:47.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:47 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi184", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T07:38:47.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:47 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:47.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:47 smithi184 ceph-mon[37288]: Reconfiguring daemon crash.smithi184 on smithi184 2024-06-06T07:38:48.963 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:48 smithi184 ceph-mon[37288]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:48.963 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:48 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:48.963 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:48 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:48.963 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:48 smithi184 ceph-mon[37288]: Reconfiguring mgr.smithi184.cqttux (monmap changed)... 2024-06-06T07:38:48.964 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:48 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.cqttux", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T07:38:48.964 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:48 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T07:38:48.964 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:48 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:48.964 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:48 smithi184 ceph-mon[37288]: Reconfiguring daemon mgr.smithi184.cqttux on smithi184 2024-06-06T07:38:49.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:48 smithi082 ceph-mon[29354]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:49.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:48 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:49.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:48 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:49.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:48 smithi082 ceph-mon[29354]: Reconfiguring mgr.smithi184.cqttux (monmap changed)... 2024-06-06T07:38:49.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:48 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.cqttux", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T07:38:49.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:48 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T07:38:49.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:48 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:49.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:48 smithi082 ceph-mon[29354]: Reconfiguring daemon mgr.smithi184.cqttux on smithi184 2024-06-06T07:38:50.194 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:50.194 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:50.195 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:50 smithi082 ceph-mon[29354]: Reconfiguring mon.smithi184 (monmap changed)... 2024-06-06T07:38:50.195 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T07:38:50.195 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T07:38:50.195 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:50.195 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:50 smithi082 ceph-mon[29354]: Reconfiguring daemon mon.smithi184 on smithi184 2024-06-06T07:38:50.195 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:50 smithi082 ceph-mon[29354]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:50.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:50 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:50.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:50 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:50.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:50 smithi184 ceph-mon[37288]: Reconfiguring mon.smithi184 (monmap changed)... 2024-06-06T07:38:50.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:50 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T07:38:50.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:50 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T07:38:50.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:50 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:38:50.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:50 smithi184 ceph-mon[37288]: Reconfiguring daemon mon.smithi184 on smithi184 2024-06-06T07:38:50.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:50 smithi184 ceph-mon[37288]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:51.383 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.383 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.383 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T07:38:51.383 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T07:38:51.383 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi082.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi082.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi082.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi082.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi082.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T07:38:51.384 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi082.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T07:38:51.385 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.385 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:38:51.385 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.385 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.433 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.433 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.433 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T07:38:51.433 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T07:38:51.433 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi082.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T07:38:51.433 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi082.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi082.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi082.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi082.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi082.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.434 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:38:51.435 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:51.435 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:52.501 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:52 smithi082 ceph-mon[29354]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:52.501 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:52.501 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:52.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:52 smithi184 ceph-mon[37288]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:52.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:52.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:53.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:38:53.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:53 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:38:53.812 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.811+0000 7fce557fa640 1 -- 172.21.15.82:0/3023596910 <== mgr.14221 v2:172.21.15.82:6800/2676832140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fce581040e0 con 0x7fce4006ae60 2024-06-06T07:38:53.812 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 0 on host 'smithi082' 2024-06-06T07:38:53.813 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fce4006ae60 msgr2=0x7fce4006d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:53.813 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fce4006ae60 0x7fce4006d320 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fce48006010 tx=0x7fce48000bc0 comp rx=0 tx=0).stop 2024-06-06T07:38:53.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58103640 msgr2=0x7fce580fecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:38:53.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58103640 0x7fce580fecc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00d6f0 tx=0x7fce4c00dbb0 comp rx=0 tx=0).stop 2024-06-06T07:38:53.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 shutdown_connections 2024-06-06T07:38:53.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fce4006ae60 0x7fce4006d320 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:53.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce58103640 0x7fce580fecc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:53.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 --2- 172.21.15.82:0/3023596910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce58100d10 0x7fce58100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:38:53.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 >> 172.21.15.82:0/3023596910 conn(0x7fce580fa700 msgr2=0x7fce580fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:38:53.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 shutdown_connections 2024-06-06T07:38:53.814 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:38:53.813+0000 7fce5ea52640 1 -- 172.21.15.82:0/3023596910 wait complete. 2024-06-06T07:38:54.482 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:54 smithi184 ceph-mon[37288]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:54.482 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:54 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:54.483 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:54 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:54.485 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:54 smithi082 ceph-mon[29354]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:54.485 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:54 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:54.485 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:54 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:54.507 DEBUG:teuthology.orchestra.run.smithi082:osd.0> sudo journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.0.service 2024-06-06T07:38:54.510 INFO:tasks.cephadm:Deploying osd.1 on smithi082 with /dev/vg_nvme/lv_3... 2024-06-06T07:38:54.510 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-06T07:38:54.939 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:38:55.491 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:55 smithi082 ceph-mon[29354]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-06T07:38:55.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:55 smithi184 ceph-mon[37288]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-06T07:38:56.480 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:56 smithi082 ceph-mon[29354]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-06T07:38:56.480 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:56 smithi082 ceph-mon[29354]: osdmap e7: 1 total, 0 up, 1 in 2024-06-06T07:38:56.480 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:56.480 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:56 smithi082 ceph-mon[29354]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-06-06T07:38:56.481 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:56 smithi082 ceph-mon[29354]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:56.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:56 smithi184 ceph-mon[37288]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-06T07:38:56.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:56 smithi184 ceph-mon[37288]: osdmap e7: 1 total, 0 up, 1 in 2024-06-06T07:38:56.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:56.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:56 smithi184 ceph-mon[37288]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-06-06T07:38:56.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:56 smithi184 ceph-mon[37288]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:57.196 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:38:57 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0[47695]: 2024-06-06T07:38:57.020+0000 7f2fc0b4b640 -1 osd.0 0 waiting for initial osdmap 2024-06-06T07:38:57.196 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:38:57 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0[47695]: 2024-06-06T07:38:57.030+0000 7f2fb917e640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T07:38:57.515 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:57 smithi082 ceph-mon[29354]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-06-06T07:38:57.515 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:57 smithi082 ceph-mon[29354]: osdmap e8: 1 total, 0 up, 1 in 2024-06-06T07:38:57.515 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:57 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:57.515 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:57 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:57.515 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:57 smithi082 ceph-mon[29354]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' 2024-06-06T07:38:57.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:57 smithi184 ceph-mon[37288]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-06-06T07:38:57.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:57 smithi184 ceph-mon[37288]: osdmap e8: 1 total, 0 up, 1 in 2024-06-06T07:38:57.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:57 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:57.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:57 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:57.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:57 smithi184 ceph-mon[37288]: from='osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521]' entity='osd.0' 2024-06-06T07:38:58.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:58 smithi082 ceph-mon[29354]: purged_snaps scrub starts 2024-06-06T07:38:58.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:58 smithi082 ceph-mon[29354]: purged_snaps scrub ok 2024-06-06T07:38:58.407 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:58.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:58 smithi082 ceph-mon[29354]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:58.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:58 smithi082 ceph-mon[29354]: osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] boot 2024-06-06T07:38:58.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:58 smithi082 ceph-mon[29354]: osdmap e9: 1 total, 1 up, 1 in 2024-06-06T07:38:58.408 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:58 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:58.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:58 smithi184 ceph-mon[37288]: purged_snaps scrub starts 2024-06-06T07:38:58.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:58 smithi184 ceph-mon[37288]: purged_snaps scrub ok 2024-06-06T07:38:58.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:58 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:58.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:58 smithi184 ceph-mon[37288]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T07:38:58.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:58 smithi184 ceph-mon[37288]: osd.0 [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] boot 2024-06-06T07:38:58.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:58 smithi184 ceph-mon[37288]: osdmap e9: 1 total, 1 up, 1 in 2024-06-06T07:38:58.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:58 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T07:38:58.735 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-06T07:38:58.829 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-06T07:38:58.843 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-06-06T07:38:58.843 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-06-06T07:38:58.843 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125092 s, 838 MB/s 2024-06-06T07:38:58.843 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-06-06T07:38:59.507 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_3 2024-06-06T07:38:59.737 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:59 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:59.737 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:59 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:59.737 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:38:59 smithi082 ceph-mon[29354]: osdmap e10: 1 total, 1 up, 1 in 2024-06-06T07:38:59.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:59 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:59.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:59 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:38:59.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:38:59 smithi184 ceph-mon[37288]: osdmap e10: 1 total, 1 up, 1 in 2024-06-06T07:39:00.760 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:00 smithi082 ceph-mon[29354]: pgmap v39: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:00.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:00 smithi184 ceph-mon[37288]: pgmap v39: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:00.958 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:39:02.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:02 smithi184 ceph-mon[37288]: pgmap v40: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:02.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:02 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:02.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:02 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:02.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:02 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:02.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:02 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:02.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:02 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:02.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:02 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:02.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:02 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:02.845 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:02 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:02.865 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:02 smithi082 ceph-mon[29354]: pgmap v40: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:02.866 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:02.866 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:02.866 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:02.866 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:02.866 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:02.866 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:02.866 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:02.867 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:03.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.177+0000 7f4b9cba1640 1 -- 172.21.15.82:0/958244556 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 msgr2=0x7f4b980fe960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:03.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.177+0000 7f4b9cba1640 1 --2- 172.21.15.82:0/958244556 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 0x7f4b980fe960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009880 tx=0x7f4b8802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:39:03.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.177+0000 7f4b9cba1640 1 -- 172.21.15.82:0/958244556 shutdown_connections 2024-06-06T07:39:03.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.177+0000 7f4b9cba1640 1 --2- 172.21.15.82:0/958244556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b98101520 0x7f4b981019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:03.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.177+0000 7f4b9cba1640 1 --2- 172.21.15.82:0/958244556 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 0x7f4b980fe960 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:03.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.177+0000 7f4b9cba1640 1 -- 172.21.15.82:0/958244556 >> 172.21.15.82:0/958244556 conn(0x7f4b980f9ec0 msgr2=0x7f4b980fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:39:03.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.177+0000 7f4b9cba1640 1 -- 172.21.15.82:0/958244556 shutdown_connections 2024-06-06T07:39:03.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.177+0000 7f4b9cba1640 1 -- 172.21.15.82:0/958244556 wait complete. 2024-06-06T07:39:03.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b9cba1640 1 Processor -- start 2024-06-06T07:39:03.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b9cba1640 1 -- start start 2024-06-06T07:39:03.181 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b9cba1640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 0x7f4b98195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:39:03.181 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b9cba1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b98101520 0x7f4b98196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:39:03.181 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b9cba1640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b98196840 con 0x7f4b980fe540 2024-06-06T07:39:03.181 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b9cba1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b981969b0 con 0x7f4b98101520 2024-06-06T07:39:03.181 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 0x7f4b98195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:39:03.181 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 0x7f4b98195dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:37896/0 (socket says 172.21.15.82:37896) 2024-06-06T07:39:03.181 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b9659c640 1 -- 172.21.15.82:0/2504734780 learned_addr learned my addr 172.21.15.82:0/2504734780 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.178+0000 7f4b8dd9b640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b98101520 0x7f4b98196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.179+0000 7f4b9659c640 1 -- 172.21.15.82:0/2504734780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b98101520 msgr2=0x7f4b98196300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.179+0000 7f4b9659c640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b98101520 0x7f4b98196300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.179+0000 7f4b9659c640 1 -- 172.21.15.82:0/2504734780 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b88009560 con 0x7f4b980fe540 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.179+0000 7f4b9659c640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 0x7f4b98195dc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b8802a9c0 tx=0x7f4b88005610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.179+0000 7f4b8ffff640 1 -- 172.21.15.82:0/2504734780 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b88004b20 con 0x7f4b980fe540 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.179+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9819b350 con 0x7f4b980fe540 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.179+0000 7f4b8ffff640 1 -- 172.21.15.82:0/2504734780 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b88004c80 con 0x7f4b980fe540 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.179+0000 7f4b8ffff640 1 -- 172.21.15.82:0/2504734780 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b88003b10 con 0x7f4b980fe540 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.179+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9819b890 con 0x7f4b980fe540 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.180+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9818fdf0 con 0x7f4b980fe540 2024-06-06T07:39:03.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.181+0000 7f4b8ffff640 1 -- 172.21.15.82:0/2504734780 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7f4b88031d20 con 0x7f4b980fe540 2024-06-06T07:39:03.183 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.181+0000 7f4b8ffff640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b8406ae70 0x7f4b8406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:39:03.183 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.181+0000 7f4b8ffff640 1 -- 172.21.15.82:0/2504734780 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f4b880aa2f0 con 0x7f4b980fe540 2024-06-06T07:39:03.183 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.181+0000 7f4b8dd9b640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b8406ae70 0x7f4b8406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:39:03.183 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.182+0000 7f4b8dd9b640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b8406ae70 0x7f4b8406d330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000abe0 tx=0x7f4b8000a2d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:39:03.186 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.186+0000 7f4b8ffff640 1 -- 172.21.15.82:0/2504734780 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4b88076410 con 0x7f4b980fe540 2024-06-06T07:39:03.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:03.374+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 --> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b9802c550 con 0x7f4b8406ae70 2024-06-06T07:39:03.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:03 smithi082 ceph-mon[29354]: Detected new or changed devices on smithi082 2024-06-06T07:39:03.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:03 smithi082 ceph-mon[29354]: Adjusting osd_memory_target on smithi082 to 11840M 2024-06-06T07:39:03.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:03 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:39:03.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:03 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:39:03.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:03 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:04.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:03 smithi184 ceph-mon[37288]: Detected new or changed devices on smithi082 2024-06-06T07:39:04.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:03 smithi184 ceph-mon[37288]: Adjusting osd_memory_target on smithi082 to 11840M 2024-06-06T07:39:04.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:03 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:39:04.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:03 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:39:04.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:03 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:04.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:04 smithi184 ceph-mon[37288]: from='client.14278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:39:04.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:04 smithi184 ceph-mon[37288]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:04.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:04.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:04.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:04 smithi082 ceph-mon[29354]: from='client.14278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:39:04.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:04 smithi082 ceph-mon[29354]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:04.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:04.945 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:06 smithi082 ceph-mon[29354]: pgmap v42: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:06 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1370485004' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d45e246-1b80-49e1-b33a-a8021b1e5013"}]: dispatch 2024-06-06T07:39:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:06 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1370485004' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d45e246-1b80-49e1-b33a-a8021b1e5013"}]': finished 2024-06-06T07:39:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:06 smithi082 ceph-mon[29354]: osdmap e11: 2 total, 1 up, 2 in 2024-06-06T07:39:06.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:06 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:06 smithi184 ceph-mon[37288]: pgmap v42: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:06 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1370485004' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d45e246-1b80-49e1-b33a-a8021b1e5013"}]: dispatch 2024-06-06T07:39:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:06 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1370485004' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d45e246-1b80-49e1-b33a-a8021b1e5013"}]': finished 2024-06-06T07:39:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:06 smithi184 ceph-mon[37288]: osdmap e11: 2 total, 1 up, 2 in 2024-06-06T07:39:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:06 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:07.765 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:07 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1796170757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:39:07.765 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:39:08.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:07 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1796170757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:39:08.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:39:09.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:08 smithi082 ceph-mon[29354]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:09.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:08 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:09.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:08 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:09.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:08 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:09.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:08 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:09.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:08 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:09.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:08 smithi184 ceph-mon[37288]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:09.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:08 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:09.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:08 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:09.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:08 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:09.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:08 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:09.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:08 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:10.710 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:10 smithi082 ceph-mon[29354]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:11.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:10 smithi184 ceph-mon[37288]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:13.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:12 smithi082 ceph-mon[29354]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:13.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:12 smithi184 ceph-mon[37288]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:13.875 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:13 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-06T07:39:13.875 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:13 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:14.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:13 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-06T07:39:14.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:13 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:14.840 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:14 smithi184 ceph-mon[37288]: Deploying daemon osd.1 on smithi082 2024-06-06T07:39:14.840 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:14 smithi184 ceph-mon[37288]: pgmap v47: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:14.962 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:14 smithi082 ceph-mon[29354]: Deploying daemon osd.1 on smithi082 2024-06-06T07:39:14.962 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:14 smithi082 ceph-mon[29354]: pgmap v47: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:16.880 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:16 smithi082 ceph-mon[29354]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:17.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:16 smithi184 ceph-mon[37288]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:18.960 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:18 smithi082 ceph-mon[29354]: pgmap v49: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:18.961 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:18.961 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:18.961 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:18 smithi184 ceph-mon[37288]: pgmap v49: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:20.903 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:20 smithi082 ceph-mon[29354]: pgmap v50: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:20.903 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:20 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:20.903 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:20 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:21.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:20 smithi184 ceph-mon[37288]: pgmap v50: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:21.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:20 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:21.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:20 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:21.729 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 1 on host 'smithi082' 2024-06-06T07:39:21.729 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.727+0000 7f4b8ffff640 1 -- 172.21.15.82:0/2504734780 <== mgr.14221 v2:172.21.15.82:6800/2676832140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4b9802c550 con 0x7f4b8406ae70 2024-06-06T07:39:21.731 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b8406ae70 msgr2=0x7f4b8406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:21.731 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b8406ae70 0x7f4b8406d330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000abe0 tx=0x7f4b8000a2d0 comp rx=0 tx=0).stop 2024-06-06T07:39:21.731 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 msgr2=0x7f4b98195dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:21.731 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 0x7f4b98195dc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b8802a9c0 tx=0x7f4b88005610 comp rx=0 tx=0).stop 2024-06-06T07:39:21.731 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 shutdown_connections 2024-06-06T07:39:21.731 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b8406ae70 0x7f4b8406d330 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:21.732 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b98101520 0x7f4b98196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:21.732 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 --2- 172.21.15.82:0/2504734780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b980fe540 0x7f4b98195dc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:21.732 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 >> 172.21.15.82:0/2504734780 conn(0x7f4b980f9ec0 msgr2=0x7f4b98104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:39:21.732 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 shutdown_connections 2024-06-06T07:39:21.732 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:21.730+0000 7f4b9cba1640 1 -- 172.21.15.82:0/2504734780 wait complete. 2024-06-06T07:39:22.480 DEBUG:teuthology.orchestra.run.smithi082:osd.1> sudo journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.1.service 2024-06-06T07:39:22.482 INFO:tasks.cephadm:Deploying osd.2 on smithi082 with /dev/vg_nvme/lv_2... 2024-06-06T07:39:22.482 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-06T07:39:22.975 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:22 smithi082 ceph-mon[29354]: pgmap v51: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:22.976 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:22.976 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:22.976 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:22 smithi082 ceph-mon[29354]: from='osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-06T07:39:22.976 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:39:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:22 smithi184 ceph-mon[37288]: pgmap v51: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:22 smithi184 ceph-mon[37288]: from='osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-06T07:39:23.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:39:23.923 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:39:24.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:23 smithi082 ceph-mon[29354]: from='osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-06T07:39:24.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:23 smithi082 ceph-mon[29354]: osdmap e12: 2 total, 1 up, 2 in 2024-06-06T07:39:24.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:23 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:24.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:23 smithi082 ceph-mon[29354]: from='osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-06-06T07:39:24.018 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:23 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1[55506]: 2024-06-06T07:39:23.731+0000 7f16341c8640 -1 osd.1 0 waiting for initial osdmap 2024-06-06T07:39:24.018 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:23 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1[55506]: 2024-06-06T07:39:23.735+0000 7f1630803640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T07:39:24.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:23 smithi184 ceph-mon[37288]: from='osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-06T07:39:24.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:23 smithi184 ceph-mon[37288]: osdmap e12: 2 total, 1 up, 2 in 2024-06-06T07:39:24.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:23 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:24.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:23 smithi184 ceph-mon[37288]: from='osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-06-06T07:39:24.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:24 smithi184 ceph-mon[37288]: pgmap v53: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:24.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:24 smithi184 ceph-mon[37288]: from='osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-06-06T07:39:24.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:24 smithi184 ceph-mon[37288]: osdmap e13: 2 total, 1 up, 2 in 2024-06-06T07:39:24.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:24 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:24.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:24 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:24.977 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:24 smithi082 ceph-mon[29354]: pgmap v53: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:24.977 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:24 smithi082 ceph-mon[29354]: from='osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-06-06T07:39:24.977 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:24 smithi082 ceph-mon[29354]: osdmap e13: 2 total, 1 up, 2 in 2024-06-06T07:39:24.978 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:24 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:24.978 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:24 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:25.918 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:25 smithi082 ceph-mon[29354]: purged_snaps scrub starts 2024-06-06T07:39:25.919 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:25 smithi082 ceph-mon[29354]: purged_snaps scrub ok 2024-06-06T07:39:25.919 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:25 smithi082 ceph-mon[29354]: osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] boot 2024-06-06T07:39:25.919 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:25 smithi082 ceph-mon[29354]: osdmap e14: 2 total, 2 up, 2 in 2024-06-06T07:39:25.919 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:25 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:26.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:25 smithi184 ceph-mon[37288]: purged_snaps scrub starts 2024-06-06T07:39:26.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:25 smithi184 ceph-mon[37288]: purged_snaps scrub ok 2024-06-06T07:39:26.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:25 smithi184 ceph-mon[37288]: osd.1 [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] boot 2024-06-06T07:39:26.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:25 smithi184 ceph-mon[37288]: osdmap e14: 2 total, 2 up, 2 in 2024-06-06T07:39:26.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:25 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T07:39:26.872 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:26 smithi082 ceph-mon[29354]: pgmap v56: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:26.872 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:26 smithi082 ceph-mon[29354]: osdmap e15: 2 total, 2 up, 2 in 2024-06-06T07:39:26.872 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:26 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:26.872 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:26 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:26.927 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-06T07:39:27.021 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-06T07:39:27.036 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-06-06T07:39:27.037 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-06-06T07:39:27.037 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131571 s, 797 MB/s 2024-06-06T07:39:27.037 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-06-06T07:39:27.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:26 smithi184 ceph-mon[37288]: pgmap v56: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-06T07:39:27.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:26 smithi184 ceph-mon[37288]: osdmap e15: 2 total, 2 up, 2 in 2024-06-06T07:39:27.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:26 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:27.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:26 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:28.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_2 2024-06-06T07:39:29.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:28 smithi082 ceph-mon[29354]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:29.051 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:39:29.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:28 smithi184 ceph-mon[37288]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:30.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:30.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:30.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:30.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:30.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:30.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:30.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:30.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:30.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:30.959 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:30 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:31.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:31.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:31.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:31.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:31.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:31.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:31.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:31.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:31.039 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:31.039 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:30 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:31.167 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.159+0000 7f2a27a80640 1 -- 172.21.15.82:0/223679177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 msgr2=0x7f2a200ff5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:31.167 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.159+0000 7f2a27a80640 1 --2- 172.21.15.82:0/223679177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 0x7f2a200ff5a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009880 tx=0x7f2a1002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:39:31.167 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.159+0000 7f2a27a80640 1 -- 172.21.15.82:0/223679177 shutdown_connections 2024-06-06T07:39:31.167 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.159+0000 7f2a27a80640 1 --2- 172.21.15.82:0/223679177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a20100380 0x7f2a20100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:31.167 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.159+0000 7f2a27a80640 1 --2- 172.21.15.82:0/223679177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 0x7f2a200ff5a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:31.167 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.159+0000 7f2a27a80640 1 -- 172.21.15.82:0/223679177 >> 172.21.15.82:0/223679177 conn(0x7f2a200fa700 msgr2=0x7f2a200fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a27a80640 1 -- 172.21.15.82:0/223679177 shutdown_connections 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a27a80640 1 -- 172.21.15.82:0/223679177 wait complete. 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a27a80640 1 Processor -- start 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a27a80640 1 -- start start 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a27a80640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 0x7f2a20103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a27a80640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a20100380 0x7f2a20101760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a27a80640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20103640 con 0x7f2a200ff180 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a27a80640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20101ca0 con 0x7f2a20100380 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a2581c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 0x7f2a20103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a2501b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a20100380 0x7f2a20101760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a2581c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 0x7f2a20103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42258/0 (socket says 172.21.15.82:42258) 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a2501b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a20100380 0x7f2a20101760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.82:45548/0 (socket says 172.21.15.82:45548) 2024-06-06T07:39:31.168 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.160+0000 7f2a2581c640 1 -- 172.21.15.82:0/1426252098 learned_addr learned my addr 172.21.15.82:0/1426252098 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.161+0000 7f2a2581c640 1 -- 172.21.15.82:0/1426252098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a20100380 msgr2=0x7f2a20101760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.161+0000 7f2a2581c640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a20100380 0x7f2a20101760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.161+0000 7f2a2581c640 1 -- 172.21.15.82:0/1426252098 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a10009560 con 0x7f2a200ff180 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.161+0000 7f2a2581c640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 0x7f2a20103070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2a1002a9c0 tx=0x7f2a100054f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.161+0000 7f2a1effd640 1 -- 172.21.15.82:0/1426252098 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a10004aa0 con 0x7f2a200ff180 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.161+0000 7f2a1effd640 1 -- 172.21.15.82:0/1426252098 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a10004c00 con 0x7f2a200ff180 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.161+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a20101f20 con 0x7f2a200ff180 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.161+0000 7f2a1effd640 1 -- 172.21.15.82:0/1426252098 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a10003a80 con 0x7f2a200ff180 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.161+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a20102410 con 0x7f2a200ff180 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.162+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2004e210 con 0x7f2a200ff180 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.163+0000 7f2a1effd640 1 -- 172.21.15.82:0/1426252098 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7f2a10003cf0 con 0x7f2a200ff180 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.163+0000 7f2a1effd640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a0806ae70 0x7f2a0806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.163+0000 7f2a1effd640 1 -- 172.21.15.82:0/1426252098 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f2a100aa2b0 con 0x7f2a200ff180 2024-06-06T07:39:31.169 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.163+0000 7f2a2501b640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a0806ae70 0x7f2a0806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:39:31.170 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.164+0000 7f2a2501b640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a0806ae70 0x7f2a0806d330 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2a20102df0 tx=0x7f2a1400b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:39:31.170 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.168+0000 7f2a1effd640 1 -- 172.21.15.82:0/1426252098 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2a10076350 con 0x7f2a200ff180 2024-06-06T07:39:31.359 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:31.359+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 --> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a20068370 con 0x7f2a0806ae70 2024-06-06T07:39:31.991 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:31 smithi082 ceph-mon[29354]: Detected new or changed devices on smithi082 2024-06-06T07:39:31.991 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:31 smithi082 ceph-mon[29354]: Adjusting osd_memory_target on smithi082 to 5920M 2024-06-06T07:39:31.991 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:31 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:39:31.992 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:31 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:39:31.992 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:31 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:32.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:31 smithi184 ceph-mon[37288]: Detected new or changed devices on smithi082 2024-06-06T07:39:32.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:31 smithi184 ceph-mon[37288]: Adjusting osd_memory_target on smithi082 to 5920M 2024-06-06T07:39:32.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:31 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:39:32.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:31 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:39:32.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:31 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:33.011 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:32 smithi082 ceph-mon[29354]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:39:33.011 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:32 smithi082 ceph-mon[29354]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:33.011 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:32 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:33.011 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:32 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:33.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:32 smithi184 ceph-mon[37288]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:39:33.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:32 smithi184 ceph-mon[37288]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:33.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:32 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:33.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:32 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:34.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:33 smithi082 ceph-mon[29354]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:34.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:33 smithi184 ceph-mon[37288]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:34.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:34 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1221686072' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83ec350-58d2-4ebc-bc70-29823dfa9320"}]: dispatch 2024-06-06T07:39:34.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:34 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1221686072' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b83ec350-58d2-4ebc-bc70-29823dfa9320"}]': finished 2024-06-06T07:39:34.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:34 smithi184 ceph-mon[37288]: osdmap e16: 3 total, 2 up, 3 in 2024-06-06T07:39:34.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:34 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:34.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:34 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3874358472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:39:35.120 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:34 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1221686072' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83ec350-58d2-4ebc-bc70-29823dfa9320"}]: dispatch 2024-06-06T07:39:35.120 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:34 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1221686072' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b83ec350-58d2-4ebc-bc70-29823dfa9320"}]': finished 2024-06-06T07:39:35.120 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:34 smithi082 ceph-mon[29354]: osdmap e16: 3 total, 2 up, 3 in 2024-06-06T07:39:35.121 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:34 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:35.121 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:34 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3874358472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:39:36.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:35 smithi082 ceph-mon[29354]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:36.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:35 smithi184 ceph-mon[37288]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:37.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:37.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:37.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:37.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:37.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:37.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:37.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:37.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:37.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:37.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:38.341 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:38 smithi082 ceph-mon[29354]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:38.342 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:39:38.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:38 smithi184 ceph-mon[37288]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:38.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:38 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:39:40.754 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:40 smithi082 ceph-mon[29354]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:41.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:40 smithi184 ceph-mon[37288]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:41.724 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-06T07:39:41.724 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:42.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-06T07:39:42.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:42.898 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:42 smithi082 ceph-mon[29354]: Deploying daemon osd.2 on smithi082 2024-06-06T07:39:42.898 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:42 smithi082 ceph-mon[29354]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:43.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:42 smithi184 ceph-mon[37288]: Deploying daemon osd.2 on smithi082 2024-06-06T07:39:43.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:42 smithi184 ceph-mon[37288]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:44.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:44 smithi184 ceph-mon[37288]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:45.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:44 smithi082 ceph-mon[29354]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:46.770 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:46 smithi082 ceph-mon[29354]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:46.770 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:46 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:46.771 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:46 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:46.771 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:46 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:46.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:46 smithi184 ceph-mon[37288]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:46.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:46 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:46.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:46 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:46.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:46 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:48.736 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:48 smithi082 ceph-mon[29354]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:49.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:48 smithi184 ceph-mon[37288]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:49.900 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.900+0000 7f2a1effd640 1 -- 172.21.15.82:0/1426252098 <== mgr.14221 v2:172.21.15.82:6800/2676832140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2a20068370 con 0x7f2a0806ae70 2024-06-06T07:39:49.901 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 2 on host 'smithi082' 2024-06-06T07:39:49.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a0806ae70 msgr2=0x7f2a0806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:49.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a0806ae70 0x7f2a0806d330 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2a20102df0 tx=0x7f2a1400b410 comp rx=0 tx=0).stop 2024-06-06T07:39:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 msgr2=0x7f2a20103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 0x7f2a20103070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2a1002a9c0 tx=0x7f2a100054f0 comp rx=0 tx=0).stop 2024-06-06T07:39:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 shutdown_connections 2024-06-06T07:39:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a0806ae70 0x7f2a0806d330 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a20100380 0x7f2a20101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 --2- 172.21.15.82:0/1426252098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a200ff180 0x7f2a20103070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 >> 172.21.15.82:0/1426252098 conn(0x7f2a200fa700 msgr2=0x7f2a20067c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:39:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 shutdown_connections 2024-06-06T07:39:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:39:49.902+0000 7f2a27a80640 1 -- 172.21.15.82:0/1426252098 wait complete. 2024-06-06T07:39:50.847 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:50 smithi082 ceph-mon[29354]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:50.847 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:50.847 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:50 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:50.847 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:50 smithi082 ceph-mon[29354]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-06T07:39:50.869 DEBUG:teuthology.orchestra.run.smithi082:osd.2> sudo journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.2.service 2024-06-06T07:39:50.872 INFO:tasks.cephadm:Deploying osd.3 on smithi184 with /dev/vg_nvme/lv_4... 2024-06-06T07:39:50.872 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-06T07:39:50.900 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:50 smithi184 ceph-mon[37288]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:50.900 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:50 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:50.900 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:50 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:50.900 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:50 smithi184 ceph-mon[37288]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-06T07:39:51.093 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi184/config 2024-06-06T07:39:52.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:51 smithi184 ceph-mon[37288]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-06T07:39:52.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:51 smithi184 ceph-mon[37288]: osdmap e17: 3 total, 2 up, 3 in 2024-06-06T07:39:52.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:51 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:52.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:51 smithi184 ceph-mon[37288]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-06-06T07:39:52.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:51 smithi184 ceph-mon[37288]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:52.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:51 smithi082 ceph-mon[29354]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-06T07:39:52.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:51 smithi082 ceph-mon[29354]: osdmap e17: 3 total, 2 up, 3 in 2024-06-06T07:39:52.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:51 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:52.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:51 smithi082 ceph-mon[29354]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-06-06T07:39:52.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:51 smithi082 ceph-mon[29354]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:52.745 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-06T07:39:52.813 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:52 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2[62830]: 2024-06-06T07:39:52.714+0000 7fee2aa41640 -1 osd.2 0 waiting for initial osdmap 2024-06-06T07:39:52.813 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:52 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2[62830]: 2024-06-06T07:39:52.723+0000 7fee23074640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T07:39:52.828 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-06T07:39:52.843 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-06-06T07:39:52.843 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-06-06T07:39:52.843 INFO:teuthology.orchestra.run.smithi184.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129882 s, 807 MB/s 2024-06-06T07:39:52.844 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 2024-06-06T07:39:52.844 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-06-06T07:39:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:52 smithi184 ceph-mon[37288]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-06-06T07:39:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:52 smithi184 ceph-mon[37288]: osdmap e18: 3 total, 2 up, 3 in 2024-06-06T07:39:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:39:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:52 smithi184 ceph-mon[37288]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' 2024-06-06T07:39:53.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:52 smithi082 ceph-mon[29354]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-06-06T07:39:53.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:52 smithi082 ceph-mon[29354]: osdmap e18: 3 total, 2 up, 3 in 2024-06-06T07:39:53.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:53.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:53.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:39:53.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:52 smithi082 ceph-mon[29354]: from='osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724]' entity='osd.2' 2024-06-06T07:39:53.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_4 2024-06-06T07:39:53.849 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi184/config 2024-06-06T07:39:54.144 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:53 smithi184 ceph-mon[37288]: purged_snaps scrub starts 2024-06-06T07:39:54.145 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:53 smithi184 ceph-mon[37288]: purged_snaps scrub ok 2024-06-06T07:39:54.145 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:53 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:54.145 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:53 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:54.145 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:53 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:54.145 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:53 smithi184 ceph-mon[37288]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:54.145 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:53 smithi184 ceph-mon[37288]: osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] boot 2024-06-06T07:39:54.145 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:53 smithi184 ceph-mon[37288]: osdmap e19: 3 total, 3 up, 3 in 2024-06-06T07:39:54.145 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:53 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:54.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:53 smithi082 ceph-mon[29354]: purged_snaps scrub starts 2024-06-06T07:39:54.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:53 smithi082 ceph-mon[29354]: purged_snaps scrub ok 2024-06-06T07:39:54.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:54.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:54.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:54.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:53 smithi082 ceph-mon[29354]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-06T07:39:54.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:53 smithi082 ceph-mon[29354]: osd.2 [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] boot 2024-06-06T07:39:54.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:53 smithi082 ceph-mon[29354]: osdmap e19: 3 total, 3 up, 3 in 2024-06-06T07:39:54.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:53 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T07:39:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.226+0000 7fd6a7c82640 1 -- 172.21.15.184:0/2452489327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a0100390 msgr2=0x7fd6a0100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.226+0000 7fd6a7c82640 1 --2- 172.21.15.184:0/2452489327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a0100390 0x7fd6a0100810 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd694009880 tx=0x7fd69402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:39:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.226+0000 7fd6a7c82640 1 -- 172.21.15.184:0/2452489327 shutdown_connections 2024-06-06T07:39:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.226+0000 7fd6a7c82640 1 --2- 172.21.15.184:0/2452489327 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a0100390 0x7fd6a0100810 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.226+0000 7fd6a7c82640 1 --2- 172.21.15.184:0/2452489327 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd6a00ff190 0x7fd6a00ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.226+0000 7fd6a7c82640 1 -- 172.21.15.184:0/2452489327 >> 172.21.15.184:0/2452489327 conn(0x7fd6a00fa750 msgr2=0x7fd6a00fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:39:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.226+0000 7fd6a7c82640 1 -- 172.21.15.184:0/2452489327 shutdown_connections 2024-06-06T07:39:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.226+0000 7fd6a7c82640 1 -- 172.21.15.184:0/2452489327 wait complete. 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a7c82640 1 Processor -- start 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a7c82640 1 -- start start 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a7c82640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a00ff190 0x7fd6a01030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a7c82640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd6a0100390 0x7fd6a0101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a7c82640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a0101cb0 con 0x7fd6a0100390 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a7c82640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a0101e20 con 0x7fd6a00ff190 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a5a1e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a00ff190 0x7fd6a01030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a5a1e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a00ff190 0x7fd6a01030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:48860/0 (socket says 172.21.15.184:48860) 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a5a1e640 1 -- 172.21.15.184:0/1661369310 learned_addr learned my addr 172.21.15.184:0/1661369310 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a521d640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd6a0100390 0x7fd6a0101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:39:55.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a5a1e640 1 -- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd6a0100390 msgr2=0x7fd6a0101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a5a1e640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd6a0100390 0x7fd6a0101770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a5a1e640 1 -- 172.21.15.184:0/1661369310 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd694009560 con 0x7fd6a00ff190 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.227+0000 7fd6a521d640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd6a0100390 0x7fd6a0101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.228+0000 7fd6a5a1e640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a00ff190 0x7fd6a01030d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00e940 tx=0x7fd69c00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.228+0000 7fd692ffd640 1 -- 172.21.15.184:0/1661369310 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd69c0096e0 con 0x7fd6a00ff190 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.228+0000 7fd692ffd640 1 -- 172.21.15.184:0/1661369310 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd69c0044f0 con 0x7fd6a00ff190 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.228+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a0102100 con 0x7fd6a00ff190 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.228+0000 7fd692ffd640 1 -- 172.21.15.184:0/1661369310 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd69c010430 con 0x7fd6a00ff190 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.228+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a019ddb0 con 0x7fd6a00ff190 2024-06-06T07:39:55.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.229+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a004e210 con 0x7fd6a00ff190 2024-06-06T07:39:55.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.229+0000 7fd692ffd640 1 -- 172.21.15.184:0/1661369310 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7fd69c00cc90 con 0x7fd6a00ff190 2024-06-06T07:39:55.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.229+0000 7fd692ffd640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd68c06ae60 0x7fd68c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:39:55.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.230+0000 7fd692ffd640 1 -- 172.21.15.184:0/1661369310 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2846+0+0 (secure 0 0 0) 0x7fd69c014070 con 0x7fd6a00ff190 2024-06-06T07:39:55.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.230+0000 7fd6a521d640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd68c06ae60 0x7fd68c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:39:55.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.230+0000 7fd6a521d640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd68c06ae60 0x7fd68c06d320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd694000c00 tx=0x7fd6940314e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:39:55.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.234+0000 7fd692ffd640 1 -- 172.21.15.184:0/1661369310 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd69c055680 con 0x7fd6a00ff190 2024-06-06T07:39:55.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:39:55.414+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 --> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6a0068ec0 con 0x7fd68c06ae60 2024-06-06T07:39:56.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:55 smithi082 ceph-mon[29354]: osdmap e20: 3 total, 3 up, 3 in 2024-06-06T07:39:56.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:55 smithi082 ceph-mon[29354]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:39:56.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:55 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:39:56.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:55 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:39:56.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:55 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:56.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:55 smithi082 ceph-mon[29354]: pgmap v77: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:39:56.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:55 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-06T07:39:56.029 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:55 smithi184 ceph-mon[37288]: osdmap e20: 3 total, 3 up, 3 in 2024-06-06T07:39:56.029 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:55 smithi184 ceph-mon[37288]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:39:56.029 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:55 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:39:56.029 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:55 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:39:56.029 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:55 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:56.029 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:55 smithi184 ceph-mon[37288]: pgmap v77: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:39:56.029 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:55 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-06T07:39:57.098 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-06T07:39:57.098 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: osdmap e21: 3 total, 3 up, 3 in 2024-06-06T07:39:57.098 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-06T07:39:57.098 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: Detected new or changed devices on smithi082 2024-06-06T07:39:57.098 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:57.098 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:57.098 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:57.098 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:57.098 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:57.099 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: Adjusting osd_memory_target on smithi082 to 3946M 2024-06-06T07:39:57.099 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:57.099 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:57.099 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:57.099 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:57.099 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:56 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:57.325 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-06T07:39:57.325 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: osdmap e21: 3 total, 3 up, 3 in 2024-06-06T07:39:57.325 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: Detected new or changed devices on smithi082 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: Adjusting osd_memory_target on smithi082 to 3946M 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:57.326 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:56 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:39:58.018 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:39:57 smithi082 sudo[67694]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T07:39:58.018 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:39:57 smithi082 sudo[67694]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:39:58.018 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:39:57 smithi082 sudo[67694]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:39:58.018 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:39:57 smithi082 sudo[67694]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:39:58.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:57 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-06T07:39:58.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:57 smithi082 ceph-mon[29354]: osdmap e22: 3 total, 3 up, 3 in 2024-06-06T07:39:58.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:57 smithi082 ceph-mon[29354]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:39:58.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:57 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/1791739527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7500c1-d624-43fe-8998-c20f90fd7406"}]: dispatch 2024-06-06T07:39:58.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:57 smithi082 ceph-mon[29354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7500c1-d624-43fe-8998-c20f90fd7406"}]: dispatch 2024-06-06T07:39:58.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:57 smithi082 ceph-mon[29354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b7500c1-d624-43fe-8998-c20f90fd7406"}]': finished 2024-06-06T07:39:58.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:57 smithi082 ceph-mon[29354]: osdmap e23: 4 total, 3 up, 4 in 2024-06-06T07:39:58.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:57 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:39:58.215 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:57 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-06T07:39:58.215 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:57 smithi184 ceph-mon[37288]: osdmap e22: 3 total, 3 up, 3 in 2024-06-06T07:39:58.215 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:57 smithi184 ceph-mon[37288]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:39:58.216 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:57 smithi184 ceph-mon[37288]: from='client.? 172.21.15.184:0/1791739527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7500c1-d624-43fe-8998-c20f90fd7406"}]: dispatch 2024-06-06T07:39:58.216 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:57 smithi184 ceph-mon[37288]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7500c1-d624-43fe-8998-c20f90fd7406"}]: dispatch 2024-06-06T07:39:58.216 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:57 smithi184 ceph-mon[37288]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b7500c1-d624-43fe-8998-c20f90fd7406"}]': finished 2024-06-06T07:39:58.216 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:57 smithi184 ceph-mon[37288]: osdmap e23: 4 total, 3 up, 4 in 2024-06-06T07:39:58.216 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:57 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:39:58.416 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67718]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T07:39:58.416 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67718]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:39:58.416 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67718]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:39:58.416 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67718]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:39:58.681 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67721]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T07:39:58.681 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67721]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:39:58.681 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67721]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:39:58.681 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67721]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:39:58.931 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67768]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T07:39:58.931 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:39:58.932 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67768]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:39:58.932 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67768]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:39:59.198 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:59.198 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/2712085523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:39:59.199 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67796]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T07:39:59.199 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67796]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:39:59.199 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67796]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:39:59.199 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:58 smithi082 sudo[67796]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:39:59.448 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67845]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T07:39:59.448 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67845]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:39:59.448 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67845]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:39:59.448 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67845]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:39:59.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:59 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:39:59.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:59 smithi184 ceph-mon[37288]: from='client.? 172.21.15.184:0/2712085523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:39:59.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67866]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T07:39:59.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67866]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:39:59.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67866]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:39:59.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67866]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:40:00.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67879]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T07:40:00.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67879]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:40:00.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67879]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:40:00.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:39:59 smithi082 sudo[67879]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:40:00.228 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:59 smithi184 sudo[41858]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T07:40:00.228 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:59 smithi184 sudo[41858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:40:00.228 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:59 smithi184 sudo[41858]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:40:00.228 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:39:59 smithi184 sudo[41858]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:40:00.243 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:00.238+0000 7fd692ffd640 1 -- 172.21.15.184:0/1661369310 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fd69c055a50 con 0x7fd6a00ff190 2024-06-06T07:40:00.377 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:00 smithi082 ceph-mon[29354]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T07:40:00.378 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:00 smithi082 ceph-mon[29354]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:00.378 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:00 smithi082 ceph-mon[29354]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T07:40:00.378 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:00 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:40:00.378 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:00 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:40:00.378 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:00 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:40:00.378 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:00 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:40:00.378 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:00 smithi082 ceph-mon[29354]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T07:40:00.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 sudo[42048]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T07:40:00.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 sudo[42048]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T07:40:00.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 sudo[42048]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T07:40:00.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 sudo[42048]: pam_unix(sudo:session): session closed for user root 2024-06-06T07:40:00.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 ceph-mon[37288]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T07:40:00.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 ceph-mon[37288]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:00.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 ceph-mon[37288]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T07:40:00.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:40:00.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:40:00.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi082"}]: dispatch 2024-06-06T07:40:00.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-06-06T07:40:00.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:00 smithi184 ceph-mon[37288]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T07:40:01.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:01 smithi082 ceph-mon[29354]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T07:40:01.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:01 smithi082 ceph-mon[29354]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T07:40:01.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:01 smithi082 ceph-mon[29354]: mgrmap e19: smithi082.kocfig(active, since 2m), standbys: smithi184.cqttux 2024-06-06T07:40:01.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:01.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:01.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:01.519 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:01.519 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:01.519 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:01.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:01 smithi184 ceph-mon[37288]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T07:40:01.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:01 smithi184 ceph-mon[37288]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T07:40:01.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:01 smithi184 ceph-mon[37288]: mgrmap e19: smithi082.kocfig(active, since 2m), standbys: smithi184.cqttux 2024-06-06T07:40:01.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:01 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:01.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:01 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:01.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:01 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:01.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:01 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:01.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:01 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:01.595 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:01 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:02.518 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:02 smithi082 ceph-mon[29354]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:02.544 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:02 smithi184 ceph-mon[37288]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:04.632 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:04 smithi184 ceph-mon[37288]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:04.632 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-06T07:40:04.632 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:04 smithi082 ceph-mon[29354]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-06T07:40:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:05.902 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:05 smithi184 ceph-mon[37288]: Deploying daemon osd.3 on smithi184 2024-06-06T07:40:06.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:05 smithi082 ceph-mon[29354]: Deploying daemon osd.3 on smithi184 2024-06-06T07:40:06.774 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:06 smithi184 ceph-mon[37288]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:07.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:06 smithi082 ceph-mon[29354]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:07.762 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:40:08.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:40:08.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:08 smithi184 ceph-mon[37288]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:09.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:08 smithi082 ceph-mon[29354]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:09.881 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:09 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:09.881 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:09 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:09.881 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:09 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:09.903 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:09 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:09.903 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:09 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:09.903 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:09 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:10.878 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:10 smithi184 ceph-mon[37288]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:11.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:10 smithi082 ceph-mon[29354]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:12.746 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.746+0000 7fd692ffd640 1 -- 172.21.15.184:0/1661369310 <== mgr.14221 v2:172.21.15.82:6800/2676832140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd6a0068ec0 con 0x7fd68c06ae60 2024-06-06T07:40:12.746 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 3 on host 'smithi184' 2024-06-06T07:40:12.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd68c06ae60 msgr2=0x7fd68c06d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:40:12.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd68c06ae60 0x7fd68c06d320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd694000c00 tx=0x7fd6940314e0 comp rx=0 tx=0).stop 2024-06-06T07:40:12.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a00ff190 msgr2=0x7fd6a01030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:40:12.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a00ff190 0x7fd6a01030d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00e940 tx=0x7fd69c00ee00 comp rx=0 tx=0).stop 2024-06-06T07:40:12.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 shutdown_connections 2024-06-06T07:40:12.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd68c06ae60 0x7fd68c06d320 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:12.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd6a0100390 0x7fd6a0101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:12.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 --2- 172.21.15.184:0/1661369310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd6a00ff190 0x7fd6a01030d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:12.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 >> 172.21.15.184:0/1661369310 conn(0x7fd6a00fa750 msgr2=0x7fd6a00687b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:40:12.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 shutdown_connections 2024-06-06T07:40:12.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:12.748+0000 7fd6a7c82640 1 -- 172.21.15.184:0/1661369310 wait complete. 2024-06-06T07:40:12.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:12 smithi184 ceph-mon[37288]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:13.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:12 smithi082 ceph-mon[29354]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:13.364 DEBUG:teuthology.orchestra.run.smithi184:osd.3> sudo journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.3.service 2024-06-06T07:40:13.366 INFO:tasks.cephadm:Deploying osd.4 on smithi184 with /dev/vg_nvme/lv_3... 2024-06-06T07:40:13.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-06T07:40:13.625 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi184/config 2024-06-06T07:40:14.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:13 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:14.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:13 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:14.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:13 smithi082 ceph-mon[29354]: from='osd.3 [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T07:40:14.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:13 smithi082 ceph-mon[29354]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T07:40:14.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:13 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:14.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:13 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:14.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:13 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:14.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:13 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:14.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:13 smithi184 ceph-mon[37288]: from='osd.3 [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T07:40:14.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:13 smithi184 ceph-mon[37288]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T07:40:14.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:13 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:14.038 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:13 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:14.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:14 smithi184 ceph-mon[37288]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:14.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:14 smithi184 ceph-mon[37288]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-06T07:40:14.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:14 smithi184 ceph-mon[37288]: osdmap e24: 4 total, 3 up, 4 in 2024-06-06T07:40:14.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:14 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:14.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:14 smithi184 ceph-mon[37288]: from='osd.3 [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:40:14.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:14 smithi184 ceph-mon[37288]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:40:14.845 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:40:14 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3[43290]: 2024-06-06T07:40:14.758+0000 7fce283df640 -1 osd.3 0 waiting for initial osdmap 2024-06-06T07:40:14.845 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:40:14 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3[43290]: 2024-06-06T07:40:14.763+0000 7fce24a1a640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T07:40:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:14 smithi082 ceph-mon[29354]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:14 smithi082 ceph-mon[29354]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-06T07:40:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:14 smithi082 ceph-mon[29354]: osdmap e24: 4 total, 3 up, 4 in 2024-06-06T07:40:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:14 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:14 smithi082 ceph-mon[29354]: from='osd.3 [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:40:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:14 smithi082 ceph-mon[29354]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:40:15.340 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-06T07:40:15.434 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-06T07:40:15.448 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-06-06T07:40:15.448 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-06-06T07:40:15.449 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012417 s, 844 MB/s 2024-06-06T07:40:15.449 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-06-06T07:40:15.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:15 smithi184 ceph-mon[37288]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-06-06T07:40:15.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:15 smithi184 ceph-mon[37288]: osdmap e25: 4 total, 3 up, 4 in 2024-06-06T07:40:15.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:15 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:15.845 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:15 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:16.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:15 smithi082 ceph-mon[29354]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-06-06T07:40:16.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:15 smithi082 ceph-mon[29354]: osdmap e25: 4 total, 3 up, 4 in 2024-06-06T07:40:16.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:15 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:16.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:15 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:16.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_3 2024-06-06T07:40:16.944 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:16 smithi184 ceph-mon[37288]: purged_snaps scrub starts 2024-06-06T07:40:16.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:16 smithi184 ceph-mon[37288]: purged_snaps scrub ok 2024-06-06T07:40:16.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:16 smithi184 ceph-mon[37288]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:16.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:16 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:16.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:16 smithi184 ceph-mon[37288]: osd.3 [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] boot 2024-06-06T07:40:16.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:16 smithi184 ceph-mon[37288]: osdmap e26: 4 total, 4 up, 4 in 2024-06-06T07:40:16.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:16 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:17.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:16 smithi082 ceph-mon[29354]: purged_snaps scrub starts 2024-06-06T07:40:17.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:16 smithi082 ceph-mon[29354]: purged_snaps scrub ok 2024-06-06T07:40:17.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:16 smithi082 ceph-mon[29354]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-06-06T07:40:17.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:16 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:17.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:16 smithi082 ceph-mon[29354]: osd.3 [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] boot 2024-06-06T07:40:17.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:16 smithi082 ceph-mon[29354]: osdmap e26: 4 total, 4 up, 4 in 2024-06-06T07:40:17.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:16 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T07:40:17.159 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi184/config 2024-06-06T07:40:17.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:17 smithi184 ceph-mon[37288]: osdmap e27: 4 total, 4 up, 4 in 2024-06-06T07:40:17.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:17 smithi184 ceph-mon[37288]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:18.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:17 smithi082 ceph-mon[29354]: osdmap e27: 4 total, 4 up, 4 in 2024-06-06T07:40:18.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:17 smithi082 ceph-mon[29354]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:18.935 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: osdmap e28: 4 total, 4 up, 4 in 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: Detected new or changed devices on smithi184 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: Adjusting osd_memory_target on smithi184 to 14912M 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:18.936 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:18.937 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:18.937 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:18 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:19.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 -- 172.21.15.184:0/4284405110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 msgr2=0x7f74c8100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:40:19.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 --2- 172.21.15.184:0/4284405110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 0x7f74c8100800 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009880 tx=0x7f74c402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:40:19.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 -- 172.21.15.184:0/4284405110 shutdown_connections 2024-06-06T07:40:19.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 --2- 172.21.15.184:0/4284405110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 0x7f74c8100800 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:19.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 --2- 172.21.15.184:0/4284405110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74c80ff180 0x7f74c80ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:19.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 -- 172.21.15.184:0/4284405110 >> 172.21.15.184:0/4284405110 conn(0x7f74c80fa720 msgr2=0x7f74c80fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:40:19.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 -- 172.21.15.184:0/4284405110 shutdown_connections 2024-06-06T07:40:19.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 -- 172.21.15.184:0/4284405110 wait complete. 2024-06-06T07:40:19.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 Processor -- start 2024-06-06T07:40:19.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.251+0000 7f74cfd13640 1 -- start start 2024-06-06T07:40:19.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cfd13640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74c80ff180 0x7f74c8103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:40:19.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cfd13640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 0x7f74c8101760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:40:19.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cfd13640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8103640 con 0x7f74c80ff180 2024-06-06T07:40:19.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cfd13640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8101ca0 con 0x7f74c8100380 2024-06-06T07:40:19.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cd2ae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 0x7f74c8101760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:40:19.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cd2ae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 0x7f74c8101760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:54212/0 (socket says 172.21.15.184:54212) 2024-06-06T07:40:19.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cd2ae640 1 -- 172.21.15.184:0/3292469295 learned_addr learned my addr 172.21.15.184:0/3292469295 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cdaaf640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74c80ff180 0x7f74c8103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cd2ae640 1 -- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74c80ff180 msgr2=0x7f74c8103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cd2ae640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74c80ff180 0x7f74c8103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cd2ae640 1 -- 172.21.15.184:0/3292469295 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c4009560 con 0x7f74c8100380 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cd2ae640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 0x7f74c8101760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006010 tx=0x7f74c40055a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74baffd640 1 -- 172.21.15.184:0/3292469295 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c4004aa0 con 0x7f74c8100380 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74baffd640 1 -- 172.21.15.184:0/3292469295 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74c4004c00 con 0x7f74c8100380 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c8101f20 con 0x7f74c8100380 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.252+0000 7f74baffd640 1 -- 172.21.15.184:0/3292469295 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c4003ac0 con 0x7f74c8100380 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.253+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c8102490 con 0x7f74c8100380 2024-06-06T07:40:19.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.254+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c8107d80 con 0x7f74c8100380 2024-06-06T07:40:19.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.254+0000 7f74baffd640 1 -- 172.21.15.184:0/3292469295 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f74c4003d30 con 0x7f74c8100380 2024-06-06T07:40:19.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.255+0000 7f74baffd640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f74b406af00 0x7f74b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:40:19.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.255+0000 7f74baffd640 1 -- 172.21.15.184:0/3292469295 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f74c40aa940 con 0x7f74c8100380 2024-06-06T07:40:19.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.255+0000 7f74cdaaf640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f74b406af00 0x7f74b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:40:19.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.255+0000 7f74cdaaf640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f74b406af00 0x7f74b406d3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f74bc005fd0 tx=0x7f74bc00b390 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:40:19.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.259+0000 7f74baffd640 1 -- 172.21.15.184:0/3292469295 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f74c40763b0 con 0x7f74c8100380 2024-06-06T07:40:19.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: osdmap e28: 4 total, 4 up, 4 in 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: Detected new or changed devices on smithi184 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: Adjusting osd_memory_target on smithi184 to 14912M 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:19.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:19.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:18 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:19.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:19.440+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 --> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f74c8068eb0 con 0x7f74b406af00 2024-06-06T07:40:19.983 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:19 smithi184 ceph-mon[37288]: osdmap e29: 4 total, 4 up, 4 in 2024-06-06T07:40:19.983 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:19 smithi184 ceph-mon[37288]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:40:19.983 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:19 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:40:19.983 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:19 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:40:19.983 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:19 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:19.983 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:19 smithi184 ceph-mon[37288]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:19.984 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:19 smithi184 ceph-mon[37288]: osdmap e30: 4 total, 4 up, 4 in 2024-06-06T07:40:20.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:19 smithi082 ceph-mon[29354]: osdmap e29: 4 total, 4 up, 4 in 2024-06-06T07:40:20.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:19 smithi082 ceph-mon[29354]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:40:20.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:19 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:40:20.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:19 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:40:20.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:19 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:20.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:19 smithi082 ceph-mon[29354]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:20.268 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:19 smithi082 ceph-mon[29354]: osdmap e30: 4 total, 4 up, 4 in 2024-06-06T07:40:21.901 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:21 smithi184 ceph-mon[37288]: from='client.? 172.21.15.184:0/11930756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84b5bb0-f47c-44c4-a092-8d5580f52b71"}]: dispatch 2024-06-06T07:40:21.901 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:21 smithi184 ceph-mon[37288]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84b5bb0-f47c-44c4-a092-8d5580f52b71"}]: dispatch 2024-06-06T07:40:21.901 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:21 smithi184 ceph-mon[37288]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f84b5bb0-f47c-44c4-a092-8d5580f52b71"}]': finished 2024-06-06T07:40:21.901 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:21 smithi184 ceph-mon[37288]: osdmap e31: 5 total, 4 up, 5 in 2024-06-06T07:40:21.901 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:21 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:22.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:21 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/11930756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84b5bb0-f47c-44c4-a092-8d5580f52b71"}]: dispatch 2024-06-06T07:40:22.035 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:21 smithi082 ceph-mon[29354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84b5bb0-f47c-44c4-a092-8d5580f52b71"}]: dispatch 2024-06-06T07:40:22.035 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:21 smithi082 ceph-mon[29354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f84b5bb0-f47c-44c4-a092-8d5580f52b71"}]': finished 2024-06-06T07:40:22.035 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:21 smithi082 ceph-mon[29354]: osdmap e31: 5 total, 4 up, 5 in 2024-06-06T07:40:22.035 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:21 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:23.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:22 smithi082 ceph-mon[29354]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-06T07:40:23.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:22 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/2827318674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:40:23.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:40:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:22 smithi184 ceph-mon[37288]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-06T07:40:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:22 smithi184 ceph-mon[37288]: from='client.? 172.21.15.184:0/2827318674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:40:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:40:25.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:24 smithi082 ceph-mon[29354]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-06-06T07:40:25.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:24 smithi184 ceph-mon[37288]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-06-06T07:40:26.664 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:26 smithi184 ceph-mon[37288]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-06-06T07:40:27.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:26 smithi082 ceph-mon[29354]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-06-06T07:40:28.863 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:28 smithi184 ceph-mon[37288]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-06T07:40:28.863 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:28 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-06T07:40:28.863 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:28 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:29.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:28 smithi082 ceph-mon[29354]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-06T07:40:29.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:28 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-06T07:40:29.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:28 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:29.761 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:29 smithi184 ceph-mon[37288]: Deploying daemon osd.4 on smithi184 2024-06-06T07:40:30.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:29 smithi082 ceph-mon[29354]: Deploying daemon osd.4 on smithi184 2024-06-06T07:40:30.758 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:30 smithi184 ceph-mon[37288]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-06-06T07:40:31.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:30 smithi082 ceph-mon[29354]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-06-06T07:40:32.746 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:32 smithi184 ceph-mon[37288]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:33.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:32 smithi082 ceph-mon[29354]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:33.901 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:33 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:33.902 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:33 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:33.902 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:33 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:34.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:33 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:34.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:33 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:34.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:33 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:34.790 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:34 smithi184 ceph-mon[37288]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:35.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:34 smithi082 ceph-mon[29354]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:36.563 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 4 on host 'smithi184' 2024-06-06T07:40:36.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.562+0000 7f74baffd640 1 -- 172.21.15.184:0/3292469295 <== mgr.14221 v2:172.21.15.82:6800/2676832140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f74c8068eb0 con 0x7f74b406af00 2024-06-06T07:40:36.565 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f74b406af00 msgr2=0x7f74b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:40:36.565 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f74b406af00 0x7f74b406d3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f74bc005fd0 tx=0x7f74bc00b390 comp rx=0 tx=0).stop 2024-06-06T07:40:36.565 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 msgr2=0x7f74c8101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:40:36.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 0x7f74c8101760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006010 tx=0x7f74c40055a0 comp rx=0 tx=0).stop 2024-06-06T07:40:36.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 shutdown_connections 2024-06-06T07:40:36.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f74b406af00 0x7f74b406d3c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:36.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74c8100380 0x7f74c8101760 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:36.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 --2- 172.21.15.184:0/3292469295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f74c80ff180 0x7f74c8103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:36.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 >> 172.21.15.184:0/3292469295 conn(0x7f74c80fa720 msgr2=0x7f74c80687a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:40:36.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 shutdown_connections 2024-06-06T07:40:36.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:36.565+0000 7f74cfd13640 1 -- 172.21.15.184:0/3292469295 wait complete. 2024-06-06T07:40:36.843 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:36 smithi184 ceph-mon[37288]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:36.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:36 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:36.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:36 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:37.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:36 smithi082 ceph-mon[29354]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:37.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:36 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:37.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:36 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:37.102 DEBUG:teuthology.orchestra.run.smithi184:osd.4> sudo journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.4.service 2024-06-06T07:40:37.105 INFO:tasks.cephadm:Deploying osd.5 on smithi184 with /dev/vg_nvme/lv_2... 2024-06-06T07:40:37.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-06T07:40:37.350 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi184/config 2024-06-06T07:40:37.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:37 smithi184 ceph-mon[37288]: from='osd.4 [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T07:40:37.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:37 smithi184 ceph-mon[37288]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T07:40:37.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:37.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:37.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:40:38.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:37 smithi082 ceph-mon[29354]: from='osd.4 [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T07:40:38.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:37 smithi082 ceph-mon[29354]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T07:40:38.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:38.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:38.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:40:38.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:38 smithi184 ceph-mon[37288]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:38.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:38 smithi184 ceph-mon[37288]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-06T07:40:38.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:38 smithi184 ceph-mon[37288]: osdmap e32: 5 total, 4 up, 5 in 2024-06-06T07:40:38.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:38 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:38.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:38 smithi184 ceph-mon[37288]: from='osd.4 [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:40:38.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:38 smithi184 ceph-mon[37288]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:40:38.844 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:40:38 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:40:38.641+0000 7f6a2cb8c640 -1 osd.4 0 waiting for initial osdmap 2024-06-06T07:40:38.845 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:40:38 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:40:38.648+0000 7f6a291c7640 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T07:40:39.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:38 smithi082 ceph-mon[29354]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:39.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:38 smithi082 ceph-mon[29354]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-06T07:40:39.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:38 smithi082 ceph-mon[29354]: osdmap e32: 5 total, 4 up, 5 in 2024-06-06T07:40:39.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:38 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:39.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:38 smithi082 ceph-mon[29354]: from='osd.4 [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:40:39.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:38 smithi082 ceph-mon[29354]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:40:39.474 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-06T07:40:39.564 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-06T07:40:39.579 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-06-06T07:40:39.580 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-06-06T07:40:39.580 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141022 s, 744 MB/s 2024-06-06T07:40:39.580 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-06-06T07:40:39.828 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:39 smithi184 ceph-mon[37288]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-06-06T07:40:39.828 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:39 smithi184 ceph-mon[37288]: osdmap e33: 5 total, 4 up, 5 in 2024-06-06T07:40:39.828 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:39 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:39.828 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:39 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:40.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:39 smithi082 ceph-mon[29354]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-06-06T07:40:40.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:39 smithi082 ceph-mon[29354]: osdmap e33: 5 total, 4 up, 5 in 2024-06-06T07:40:40.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:39 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:40.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:39 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:40.852 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:40 smithi184 ceph-mon[37288]: purged_snaps scrub starts 2024-06-06T07:40:40.852 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:40 smithi184 ceph-mon[37288]: purged_snaps scrub ok 2024-06-06T07:40:40.852 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:40 smithi184 ceph-mon[37288]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:40.852 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:40 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:40.852 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:40 smithi184 ceph-mon[37288]: osd.4 [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] boot 2024-06-06T07:40:40.852 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:40 smithi184 ceph-mon[37288]: osdmap e34: 5 total, 5 up, 5 in 2024-06-06T07:40:40.852 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:40 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:40.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi184:vg_nvme/lv_2 2024-06-06T07:40:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:40 smithi082 ceph-mon[29354]: purged_snaps scrub starts 2024-06-06T07:40:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:40 smithi082 ceph-mon[29354]: purged_snaps scrub ok 2024-06-06T07:40:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:40 smithi082 ceph-mon[29354]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-06-06T07:40:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:40 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:40 smithi082 ceph-mon[29354]: osd.4 [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] boot 2024-06-06T07:40:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:40 smithi082 ceph-mon[29354]: osdmap e34: 5 total, 5 up, 5 in 2024-06-06T07:40:41.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:40 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T07:40:41.603 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi184/config 2024-06-06T07:40:41.725 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: osdmap e35: 5 total, 5 up, 5 in 2024-06-06T07:40:41.725 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:41.725 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:41.725 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:40:41.725 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:40:41.725 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:41.726 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:41.726 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:41.726 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:41.726 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:41.726 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:41.726 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:41.726 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:41 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:42.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: osdmap e35: 5 total, 5 up, 5 in 2024-06-06T07:40:42.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:42.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:42.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:40:42.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:40:42.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:42.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:42.021 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:42.021 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:42.021 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:42.022 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:42.022 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:40:42.022 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:41 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:42.763 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:42 smithi184 ceph-mon[37288]: Detected new or changed devices on smithi184 2024-06-06T07:40:42.763 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:42 smithi184 ceph-mon[37288]: Adjusting osd_memory_target on smithi184 to 7456M 2024-06-06T07:40:42.763 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:42 smithi184 ceph-mon[37288]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:42.965 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.964+0000 7f8ff1941640 1 -- 172.21.15.184:0/9816655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 msgr2=0x7f8fec100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:40:42.965 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.964+0000 7f8ff1941640 1 --2- 172.21.15.184:0/9816655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 0x7f8fec100810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009880 tx=0x7f8fd402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:40:42.965 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.964+0000 7f8ff1941640 1 -- 172.21.15.184:0/9816655 shutdown_connections 2024-06-06T07:40:42.965 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.964+0000 7f8ff1941640 1 --2- 172.21.15.184:0/9816655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 0x7f8fec100810 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:42.965 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.964+0000 7f8ff1941640 1 --2- 172.21.15.184:0/9816655 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8fec0ff190 0x7f8fec0ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:42.966 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.964+0000 7f8ff1941640 1 -- 172.21.15.184:0/9816655 >> 172.21.15.184:0/9816655 conn(0x7f8fec0fa750 msgr2=0x7f8fec0fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:40:42.966 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.964+0000 7f8ff1941640 1 -- 172.21.15.184:0/9816655 shutdown_connections 2024-06-06T07:40:42.966 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.965+0000 7f8ff1941640 1 -- 172.21.15.184:0/9816655 wait complete. 2024-06-06T07:40:42.968 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.965+0000 7f8ff1941640 1 Processor -- start 2024-06-06T07:40:42.968 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.965+0000 7f8ff1941640 1 -- start start 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.965+0000 7f8ff1941640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8fec0ff190 0x7f8fec195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.965+0000 7f8ff1941640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 0x7f8fec1962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.965+0000 7f8ff1941640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec196870 con 0x7f8fec0ff190 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.965+0000 7f8ff1941640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec1969e0 con 0x7f8fec100390 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8fe3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 0x7f8fec1962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8fe3fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 0x7f8fec1962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:50748/0 (socket says 172.21.15.184:50748) 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8fe3fff640 1 -- 172.21.15.184:0/2165838551 learned_addr learned my addr 172.21.15.184:0/2165838551 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8feaffd640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8fec0ff190 0x7f8fec195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8fe3fff640 1 -- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8fec0ff190 msgr2=0x7f8fec195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8fe3fff640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8fec0ff190 0x7f8fec195d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8fe3fff640 1 -- 172.21.15.184:0/2165838551 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd4009560 con 0x7f8fec100390 2024-06-06T07:40:42.969 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8feaffd640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8fec0ff190 0x7f8fec195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8fe3fff640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 0x7f8fec1962a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4005610 tx=0x7f8fd4005640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8ff093f640 1 -- 172.21.15.184:0/2165838551 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fd4004ba0 con 0x7f8fec100390 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8ff093f640 1 -- 172.21.15.184:0/2165838551 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fd4004d00 con 0x7f8fec100390 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fec19b3c0 con 0x7f8fec100390 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8ff093f640 1 -- 172.21.15.184:0/2165838551 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fd4003bd0 con 0x7f8fec100390 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.966+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fec19b930 con 0x7f8fec100390 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.967+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fec107d90 con 0x7f8fec100390 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.968+0000 7f8ff093f640 1 -- 172.21.15.184:0/2165838551 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f8fd4031640 con 0x7f8fec100390 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.968+0000 7f8ff093f640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8fd806aef0 0x7f8fd806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.968+0000 7f8ff093f640 1 -- 172.21.15.184:0/2165838551 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f8fd40aae00 con 0x7f8fec100390 2024-06-06T07:40:42.970 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.968+0000 7f8feaffd640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8fd806aef0 0x7f8fd806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:40:42.971 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.969+0000 7f8feaffd640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8fd806aef0 0x7f8fd806d3b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0058c0 tx=0x7f8fdc005d70 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:40:42.972 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:42.972+0000 7f8ff093f640 1 -- 172.21.15.184:0/2165838551 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8fd4076590 con 0x7f8fec100390 2024-06-06T07:40:43.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:42 smithi082 ceph-mon[29354]: Detected new or changed devices on smithi184 2024-06-06T07:40:43.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:42 smithi082 ceph-mon[29354]: Adjusting osd_memory_target on smithi184 to 7456M 2024-06-06T07:40:43.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:42 smithi082 ceph-mon[29354]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:43.154 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:40:43.154+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 --> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fec068e30 con 0x7f8fd806aef0 2024-06-06T07:40:43.846 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:43 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:40:43.846 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:43 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:40:43.846 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:43 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:44.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:43 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T07:40:44.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:43 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T07:40:44.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:43 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:44.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:44 smithi184 ceph-mon[37288]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:40:44.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:44 smithi184 ceph-mon[37288]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:45.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:44 smithi082 ceph-mon[29354]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:40:45.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:44 smithi082 ceph-mon[29354]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:45.783 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:45 smithi184 ceph-mon[37288]: from='client.? 172.21.15.184:0/691966018' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c6802b9-a205-4b2c-bd29-db1d2e8e80a3"}]: dispatch 2024-06-06T07:40:45.783 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:45 smithi184 ceph-mon[37288]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c6802b9-a205-4b2c-bd29-db1d2e8e80a3"}]: dispatch 2024-06-06T07:40:45.783 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:45 smithi184 ceph-mon[37288]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c6802b9-a205-4b2c-bd29-db1d2e8e80a3"}]': finished 2024-06-06T07:40:45.783 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:45 smithi184 ceph-mon[37288]: osdmap e36: 6 total, 5 up, 6 in 2024-06-06T07:40:45.783 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:45 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:40:46.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:45 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/691966018' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c6802b9-a205-4b2c-bd29-db1d2e8e80a3"}]: dispatch 2024-06-06T07:40:46.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:45 smithi082 ceph-mon[29354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c6802b9-a205-4b2c-bd29-db1d2e8e80a3"}]: dispatch 2024-06-06T07:40:46.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:45 smithi082 ceph-mon[29354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c6802b9-a205-4b2c-bd29-db1d2e8e80a3"}]': finished 2024-06-06T07:40:46.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:45 smithi082 ceph-mon[29354]: osdmap e36: 6 total, 5 up, 6 in 2024-06-06T07:40:46.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:45 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:40:47.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:46 smithi082 ceph-mon[29354]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:47.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:46 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/4095935248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:40:47.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:46 smithi184 ceph-mon[37288]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:47.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:46 smithi184 ceph-mon[37288]: from='client.? 172.21.15.184:0/4095935248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T07:40:48.950 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:48 smithi184 ceph-mon[37288]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:49.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:48 smithi082 ceph-mon[29354]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:50.777 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:50 smithi184 ceph-mon[37288]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:51.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:50 smithi082 ceph-mon[29354]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:52.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:52 smithi184 ceph-mon[37288]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:52.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-06T07:40:52.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:52.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:40:53.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:52 smithi082 ceph-mon[29354]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:53.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-06T07:40:53.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:40:53.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:40:53.759 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:53 smithi184 ceph-mon[37288]: Deploying daemon osd.5 on smithi184 2024-06-06T07:40:54.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:53 smithi082 ceph-mon[29354]: Deploying daemon osd.5 on smithi184 2024-06-06T07:40:54.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:54 smithi184 ceph-mon[37288]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:55.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:54 smithi082 ceph-mon[29354]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:56.818 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:56 smithi184 ceph-mon[37288]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:57.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:56 smithi082 ceph-mon[29354]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:57.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:57 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:57.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:57 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:57.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:57 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:58.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:57 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:40:58.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:57 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:58.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:57 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:40:59.000 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:40:58 smithi184 ceph-mon[37288]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:40:59.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:40:58 smithi082 ceph-mon[29354]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:41:00.374 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.373+0000 7f8ff093f640 1 -- 172.21.15.184:0/2165838551 <== mgr.14221 v2:172.21.15.82:6800/2676832140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8fec068e30 con 0x7f8fd806aef0 2024-06-06T07:41:00.375 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 5 on host 'smithi184' 2024-06-06T07:41:00.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8fd806aef0 msgr2=0x7f8fd806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:00.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8fd806aef0 0x7f8fd806d3b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0058c0 tx=0x7f8fdc005d70 comp rx=0 tx=0).stop 2024-06-06T07:41:00.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 msgr2=0x7f8fec1962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:00.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 0x7f8fec1962a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4005610 tx=0x7f8fd4005640 comp rx=0 tx=0).stop 2024-06-06T07:41:00.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 shutdown_connections 2024-06-06T07:41:00.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8fd806aef0 0x7f8fd806d3b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:00.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fec100390 0x7f8fec1962a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:00.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 --2- 172.21.15.184:0/2165838551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8fec0ff190 0x7f8fec195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:00.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 >> 172.21.15.184:0/2165838551 conn(0x7f8fec0fa750 msgr2=0x7f8fec068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:00.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 shutdown_connections 2024-06-06T07:41:00.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:00.376+0000 7f8ff1941640 1 -- 172.21.15.184:0/2165838551 wait complete. 2024-06-06T07:41:00.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:00 smithi082 ceph-mon[29354]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:41:00.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:00 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:00.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:00 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:00.904 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:00 smithi184 ceph-mon[37288]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:41:00.904 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:00 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:00.904 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:00 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:00.925 DEBUG:teuthology.orchestra.run.smithi184:osd.5> sudo journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.5.service 2024-06-06T07:41:00.927 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-06-06T07:41:00.927 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-06T07:41:01.158 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:01.887 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:01 smithi082 ceph-mon[29354]: from='osd.5 [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T07:41:01.887 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:01 smithi082 ceph-mon[29354]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T07:41:01.887 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:01.887 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:01 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:01.958 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:01 smithi184 ceph-mon[37288]: from='osd.5 [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T07:41:01.958 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:01 smithi184 ceph-mon[37288]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T07:41:01.958 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:01 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:01.958 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:01 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:02.822 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:02 smithi082 ceph-mon[29354]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:41:02.822 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:02 smithi082 ceph-mon[29354]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-06T07:41:02.822 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:02 smithi082 ceph-mon[29354]: osdmap e37: 6 total, 5 up, 6 in 2024-06-06T07:41:02.822 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:02 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:02.822 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:02 smithi082 ceph-mon[29354]: from='osd.5 [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:41:02.823 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:02 smithi082 ceph-mon[29354]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:41:02.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.901+0000 7fc5973cc640 1 -- 172.21.15.82:0/1759394367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590100d10 msgr2=0x7fc590103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:02.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.901+0000 7fc5973cc640 1 --2- 172.21.15.82:0/1759394367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590100d10 0x7fc590103100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc588009c20 tx=0x7fc58802a970 comp rx=0 tx=0).stop 2024-06-06T07:41:02.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.901+0000 7fc5973cc640 1 -- 172.21.15.82:0/1759394367 shutdown_connections 2024-06-06T07:41:02.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 --2- 172.21.15.82:0/1759394367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590103640 0x7fc590105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:02.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 --2- 172.21.15.82:0/1759394367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590100d10 0x7fc590103100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:02.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 -- 172.21.15.82:0/1759394367 >> 172.21.15.82:0/1759394367 conn(0x7fc5900fa700 msgr2=0x7fc5900fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:02.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 -- 172.21.15.82:0/1759394367 shutdown_connections 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 -- 172.21.15.82:0/1759394367 wait complete. 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 Processor -- start 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 -- start start 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590100d10 0x7fc590195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590103640 0x7fc5901962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc590196890 con 0x7fc590103640 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc5973cc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc590196a00 con 0x7fc590100d10 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc594967640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590103640 0x7fc5901962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc594967640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590103640 0x7fc5901962c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:49750/0 (socket says 172.21.15.82:49750) 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.902+0000 7fc594967640 1 -- 172.21.15.82:0/2172243285 learned_addr learned my addr 172.21.15.82:0/2172243285 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc595168640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590100d10 0x7fc590195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc594967640 1 -- 172.21.15.82:0/2172243285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590100d10 msgr2=0x7fc590195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc594967640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590100d10 0x7fc590195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc594967640 1 -- 172.21.15.82:0/2172243285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588009900 con 0x7fc590103640 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc594967640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590103640 0x7fc5901962c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc58400d6f0 tx=0x7fc58400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc58e7fc640 1 -- 172.21.15.82:0/2172243285 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5840051f0 con 0x7fc590103640 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc58e7fc640 1 -- 172.21.15.82:0/2172243285 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc584009ce0 con 0x7fc590103640 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59019b450 con 0x7fc590103640 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc58e7fc640 1 -- 172.21.15.82:0/2172243285 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc584004e80 con 0x7fc590103640 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.903+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59019ba20 con 0x7fc590103640 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.904+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc590108e90 con 0x7fc590103640 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.904+0000 7fc58e7fc640 1 -- 172.21.15.82:0/2172243285 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fc5840046c0 con 0x7fc590103640 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.905+0000 7fc58e7fc640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc57806aef0 0x7fc57806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.905+0000 7fc58e7fc640 1 -- 172.21.15.82:0/2172243285 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4358+0+0 (secure 0 0 0) 0x7fc5840893d0 con 0x7fc590103640 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.905+0000 7fc595168640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc57806aef0 0x7fc57806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:02.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.905+0000 7fc595168640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc57806aef0 0x7fc57806d3b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc588004af0 tx=0x7fc58803d040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:02.910 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:02.910+0000 7fc58e7fc640 1 -- 172.21.15.82:0/2172243285 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc584054bf0 con 0x7fc590103640 2024-06-06T07:41:03.083 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.083+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fc590109070 con 0x7fc590103640 2024-06-06T07:41:03.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.083+0000 7fc58e7fc640 1 -- 172.21.15.82:0/2172243285 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7fc5840170a0 con 0x7fc590103640 2024-06-06T07:41:03.084 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:41:03.086 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc57806aef0 msgr2=0x7fc57806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc57806aef0 0x7fc57806d3b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc588004af0 tx=0x7fc58803d040 comp rx=0 tx=0).stop 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590103640 msgr2=0x7fc5901962c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590103640 0x7fc5901962c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc58400d6f0 tx=0x7fc58400dbb0 comp rx=0 tx=0).stop 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 shutdown_connections 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc57806aef0 0x7fc57806d3b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc590103640 0x7fc5901962c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 --2- 172.21.15.82:0/2172243285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc590100d10 0x7fc590195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 >> 172.21.15.82:0/2172243285 conn(0x7fc5900fa700 msgr2=0x7fc5900fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 shutdown_connections 2024-06-06T07:41:03.087 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:03.086+0000 7fc5973cc640 1 -- 172.21.15.82:0/2172243285 wait complete. 2024-06-06T07:41:03.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:02 smithi184 ceph-mon[37288]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:41:03.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:02 smithi184 ceph-mon[37288]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-06T07:41:03.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:02 smithi184 ceph-mon[37288]: osdmap e37: 6 total, 5 up, 6 in 2024-06-06T07:41:03.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:02 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:03.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:02 smithi184 ceph-mon[37288]: from='osd.5 [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:41:03.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:02 smithi184 ceph-mon[37288]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-06-06T07:41:03.772 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1717659639,"num_in_osds":6,"osd_in_since":1717659645,"num_remapped_pgs":0} 2024-06-06T07:41:03.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:03 smithi184 ceph-mon[37288]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-06-06T07:41:03.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:03 smithi184 ceph-mon[37288]: osdmap e38: 6 total, 5 up, 6 in 2024-06-06T07:41:03.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:03 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:03.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:03 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:03.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:03 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2172243285' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T07:41:03.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:03 smithi184 ceph-mon[37288]: from='osd.5 ' entity='osd.5' 2024-06-06T07:41:03.845 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:41:03 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:41:03.551+0000 7f3575534640 -1 osd.5 0 waiting for initial osdmap 2024-06-06T07:41:03.845 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:41:03 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:41:03.561+0000 7f356db67640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T07:41:04.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:03 smithi082 ceph-mon[29354]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-06-06T07:41:04.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:03 smithi082 ceph-mon[29354]: osdmap e38: 6 total, 5 up, 6 in 2024-06-06T07:41:04.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:03 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:04.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:03 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:04.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:03 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2172243285' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T07:41:04.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:03 smithi082 ceph-mon[29354]: from='osd.5 ' entity='osd.5' 2024-06-06T07:41:04.773 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-06T07:41:04.974 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: purged_snaps scrub starts 2024-06-06T07:41:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: purged_snaps scrub ok 2024-06-06T07:41:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:41:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:41:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:41:05.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: osd.5 [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] boot 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: osdmap e39: 6 total, 6 up, 6 in 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:41:05.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:04 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:05.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: purged_snaps scrub starts 2024-06-06T07:41:05.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: purged_snaps scrub ok 2024-06-06T07:41:05.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-06-06T07:41:05.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:05.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:05.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: osd.5 [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] boot 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: osdmap e39: 6 total, 6 up, 6 in 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:41:05.095 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:04 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:41:06.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:05 smithi082 ceph-mon[29354]: Detected new or changed devices on smithi184 2024-06-06T07:41:06.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:05 smithi082 ceph-mon[29354]: Adjusting osd_memory_target on smithi184 to 4970M 2024-06-06T07:41:06.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:05 smithi184 ceph-mon[37288]: Detected new or changed devices on smithi184 2024-06-06T07:41:06.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:05 smithi184 ceph-mon[37288]: Adjusting osd_memory_target on smithi184 to 4970M 2024-06-06T07:41:06.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.646+0000 7f61988a4640 1 -- 172.21.15.82:0/3167021233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 msgr2=0x7f61940ff5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.646+0000 7f61988a4640 1 --2- 172.21.15.82:0/3167021233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 0x7f61940ff5a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6184009880 tx=0x7f618402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.646+0000 7f61988a4640 1 -- 172.21.15.82:0/3167021233 shutdown_connections 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.646+0000 7f61988a4640 1 --2- 172.21.15.82:0/3167021233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6194100380 0x7f6194100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.646+0000 7f61988a4640 1 --2- 172.21.15.82:0/3167021233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 0x7f61940ff5a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.646+0000 7f61988a4640 1 -- 172.21.15.82:0/3167021233 >> 172.21.15.82:0/3167021233 conn(0x7f61940fa720 msgr2=0x7f61940fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.646+0000 7f61988a4640 1 -- 172.21.15.82:0/3167021233 shutdown_connections 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.646+0000 7f61988a4640 1 -- 172.21.15.82:0/3167021233 wait complete. 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f61988a4640 1 Processor -- start 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f61988a4640 1 -- start start 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f61988a4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 0x7f6194068fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:06.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f61988a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6194100380 0x7f619419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f61988a4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6194196880 con 0x7f61940ff180 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f61988a4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619419d3e0 con 0x7f6194100380 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f619259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 0x7f6194068fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f619259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 0x7f6194068fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:37144/0 (socket says 172.21.15.82:37144) 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f619259c640 1 -- 172.21.15.82:0/4064457331 learned_addr learned my addr 172.21.15.82:0/4064457331 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f618bfff640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6194100380 0x7f619419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f619259c640 1 -- 172.21.15.82:0/4064457331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6194100380 msgr2=0x7f619419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f619259c640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6194100380 0x7f619419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.647+0000 7f619259c640 1 -- 172.21.15.82:0/4064457331 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6184009560 con 0x7f61940ff180 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.648+0000 7f619259c640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 0x7f6194068fd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6184005610 tx=0x7f6184005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.648+0000 7f618b7fe640 1 -- 172.21.15.82:0/4064457331 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6184004ba0 con 0x7f61940ff180 2024-06-06T07:41:06.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.648+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619419d580 con 0x7f61940ff180 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.648+0000 7f618b7fe640 1 -- 172.21.15.82:0/4064457331 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6184004d00 con 0x7f61940ff180 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.648+0000 7f618b7fe640 1 -- 172.21.15.82:0/4064457331 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6184003bd0 con 0x7f61940ff180 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.648+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619419dac0 con 0x7f61940ff180 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.649+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619404e210 con 0x7f61940ff180 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.649+0000 7f618b7fe640 1 -- 172.21.15.82:0/4064457331 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f6184031640 con 0x7f61940ff180 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.650+0000 7f618b7fe640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f618006aef0 0x7f618006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.650+0000 7f618b7fe640 1 -- 172.21.15.82:0/4064457331 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f61840aaf30 con 0x7f61940ff180 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.650+0000 7f618bfff640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f618006aef0 0x7f618006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.650+0000 7f618bfff640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f618006aef0 0x7f618006d3b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f617c005e10 tx=0x7f617c005da0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:06.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.654+0000 7f618b7fe640 1 -- 172.21.15.82:0/4064457331 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f61840de900 con 0x7f61940ff180 2024-06-06T07:41:06.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.830+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f619418ecf0 con 0x7f61940ff180 2024-06-06T07:41:06.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.830+0000 7f618b7fe640 1 -- 172.21.15.82:0/4064457331 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f618404a0a0 con 0x7f61940ff180 2024-06-06T07:41:06.832 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:41:06.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f618006aef0 msgr2=0x7f618006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f618006aef0 0x7f618006d3b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f617c005e10 tx=0x7f617c005da0 comp rx=0 tx=0).stop 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 msgr2=0x7f6194068fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 0x7f6194068fd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6184005610 tx=0x7f6184005640 comp rx=0 tx=0).stop 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 shutdown_connections 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f618006aef0 0x7f618006d3b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6194100380 0x7f619419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 --2- 172.21.15.82:0/4064457331 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61940ff180 0x7f6194068fd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 >> 172.21.15.82:0/4064457331 conn(0x7f61940fa720 msgr2=0x7f61941039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 shutdown_connections 2024-06-06T07:41:06.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:06.832+0000 7f61988a4640 1 -- 172.21.15.82:0/4064457331 wait complete. 2024-06-06T07:41:07.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:06 smithi082 ceph-mon[29354]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:07.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:06 smithi082 ceph-mon[29354]: osdmap e40: 6 total, 6 up, 6 in 2024-06-06T07:41:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:06 smithi184 ceph-mon[37288]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:06 smithi184 ceph-mon[37288]: osdmap e40: 6 total, 6 up, 6 in 2024-06-06T07:41:07.669 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1717659664,"num_in_osds":6,"osd_in_since":1717659645,"num_remapped_pgs":0} 2024-06-06T07:41:07.669 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-06T07:41:07.874 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:07.995 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:07 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/4064457331' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T07:41:07.996 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:41:08.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:07 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/4064457331' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T07:41:08.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:41:09.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:08 smithi082 ceph-mon[29354]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:09.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:08 smithi184 ceph-mon[37288]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:09.597 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.595+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1612980095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e40fe770 msgr2=0x7fe4e40febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:09.598 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.595+0000 7fe4e93b1640 1 --2- 172.21.15.82:0/1612980095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e40fe770 0x7fe4e40febf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009880 tx=0x7fe4cc02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:09.598 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.596+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1612980095 shutdown_connections 2024-06-06T07:41:09.598 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.596+0000 7fe4e93b1640 1 --2- 172.21.15.82:0/1612980095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e40fe770 0x7fe4e40febf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:09.598 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.596+0000 7fe4e93b1640 1 --2- 172.21.15.82:0/1612980095 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4e4105240 0x7fe4e4105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:09.598 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.596+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1612980095 >> 172.21.15.82:0/1612980095 conn(0x7fe4e4075570 msgr2=0x7fe4e40fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:09.598 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.596+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1612980095 shutdown_connections 2024-06-06T07:41:09.598 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.596+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1612980095 wait complete. 2024-06-06T07:41:09.599 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.596+0000 7fe4e93b1640 1 Processor -- start 2024-06-06T07:41:09.599 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.596+0000 7fe4e93b1640 1 -- start start 2024-06-06T07:41:09.601 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e93b1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4e40fe770 0x7fe4e4196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:09.601 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e93b1640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e4105240 0x7fe4e4196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:09.601 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e93b1640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e4196f10 con 0x7fe4e4105240 2024-06-06T07:41:09.601 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e93b1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e419ac30 con 0x7fe4e40fe770 2024-06-06T07:41:09.601 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e27fc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e4105240 0x7fe4e4196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:09.601 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e27fc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e4105240 0x7fe4e4196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:37162/0 (socket says 172.21.15.82:37162) 2024-06-06T07:41:09.601 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e27fc640 1 -- 172.21.15.82:0/1237385250 learned_addr learned my addr 172.21.15.82:0/1237385250 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e2ffd640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4e40fe770 0x7fe4e4196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e27fc640 1 -- 172.21.15.82:0/1237385250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4e40fe770 msgr2=0x7fe4e4196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e27fc640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4e40fe770 0x7fe4e4196340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e27fc640 1 -- 172.21.15.82:0/1237385250 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc009560 con 0x7fe4e4105240 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e27fc640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e4105240 0x7fe4e4196880 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc02a9c0 tx=0x7fe4cc0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4dbfff640 1 -- 172.21.15.82:0/1237385250 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4cc004aa0 con 0x7fe4e4105240 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e419aeb0 con 0x7fe4e4105240 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4dbfff640 1 -- 172.21.15.82:0/1237385250 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4cc004c00 con 0x7fe4e4105240 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4dbfff640 1 -- 172.21.15.82:0/1237385250 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4cc003ab0 con 0x7fe4e4105240 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.597+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e419b3a0 con 0x7fe4e4105240 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.598+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e4101a90 con 0x7fe4e4105240 2024-06-06T07:41:09.602 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.599+0000 7fe4dbfff640 1 -- 172.21.15.82:0/1237385250 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fe4cc031d20 con 0x7fe4e4105240 2024-06-06T07:41:09.603 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.599+0000 7fe4dbfff640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe4d006aef0 0x7fe4d006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:09.603 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.599+0000 7fe4dbfff640 1 -- 172.21.15.82:0/1237385250 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe4cc0aad10 con 0x7fe4e4105240 2024-06-06T07:41:09.603 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.599+0000 7fe4e2ffd640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe4d006aef0 0x7fe4d006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:09.603 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.600+0000 7fe4e2ffd640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe4d006aef0 0x7fe4d006d3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc007dc0 tx=0x7fe4dc0072f0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:09.605 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.604+0000 7fe4dbfff640 1 -- 172.21.15.82:0/1237385250 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe4cc0762f0 con 0x7fe4e4105240 2024-06-06T07:41:09.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.783+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4e404e210 con 0x7fe4e4105240 2024-06-06T07:41:09.785 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.783+0000 7fe4dbfff640 1 -- 172.21.15.82:0/1237385250 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11336 (secure 0 0 0) 0x7fe4cc079910 con 0x7fe4e4105240 2024-06-06T07:41:09.785 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:41:09.785 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":40,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","created":"2024-06-06T07:35:28.756873+0000","modified":"2024-06-06T07:41:05.704570+0000","last_up_change":"2024-06-06T07:41:04.555363+0000","last_in_change":"2024-06-06T07:40:45.206953+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-06T07:39:55.812493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"919724ed-ba2c-4df8-aa46-cd599dcf0647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6803","nonce":879999521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6805","nonce":879999521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6809","nonce":879999521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6807","nonce":879999521}]},"public_addr":"172.21.15.82:6803/879999521","cluster_addr":"172.21.15.82:6805/879999521","heartbeat_back_addr":"172.21.15.82:6809/879999521","heartbeat_front_addr":"172.21.15.82:6807/879999521","state":["exists","up"]},{"osd":1,"uuid":"6d45e246-1b80-49e1-b33a-a8021b1e5013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6811","nonce":1482740538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6813","nonce":1482740538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6817","nonce":1482740538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6815","nonce":1482740538}]},"public_addr":"172.21.15.82:6811/1482740538","cluster_addr":"172.21.15.82:6813/1482740538","heartbeat_back_addr":"172.21.15.82:6817/1482740538","heartbeat_front_addr":"172.21.15.82:6815/1482740538","state":["exists","up"]},{"osd":2,"uuid":"b83ec350-58d2-4ebc-bc70-29823dfa9320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6819","nonce":1581462724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6821","nonce":1581462724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6825","nonce":1581462724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6823","nonce":1581462724}]},"public_addr":"172.21.15.82:6819/1581462724","cluster_addr":"172.21.15.82:6821/1581462724","heartbeat_back_addr":"172.21.15.82:6825/1581462724","heartbeat_front_addr":"172.21.15.82:6823/1581462724","state":["exists","up"]},{"osd":3,"uuid":"4b7500c1-d624-43fe-8998-c20f90fd7406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6801","nonce":3221286727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6803","nonce":3221286727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6807","nonce":3221286727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6805","nonce":3221286727}]},"public_addr":"172.21.15.184:6801/3221286727","cluster_addr":"172.21.15.184:6803/3221286727","heartbeat_back_addr":"172.21.15.184:6807/3221286727","heartbeat_front_addr":"172.21.15.184:6805/3221286727","state":["exists","up"]},{"osd":4,"uuid":"f84b5bb0-f47c-44c4-a092-8d5580f52b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6809","nonce":2263803443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6811","nonce":2263803443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6815","nonce":2263803443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6813","nonce":2263803443}]},"public_addr":"172.21.15.184:6809/2263803443","cluster_addr":"172.21.15.184:6811/2263803443","heartbeat_back_addr":"172.21.15.184:6815/2263803443","heartbeat_front_addr":"172.21.15.184:6813/2263803443","state":["exists","up"]},{"osd":5,"uuid":"4c6802b9-a205-4b2c-bd29-db1d2e8e80a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6817","nonce":2823055081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6819","nonce":2823055081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6823","nonce":2823055081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6821","nonce":2823055081}]},"public_addr":"172.21.15.184:6817/2823055081","cluster_addr":"172.21.15.184:6819/2823055081","heartbeat_back_addr":"172.21.15.184:6823/2823055081","heartbeat_front_addr":"172.21.15.184:6821/2823055081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:38:55.385571+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:39:23.464338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:39:51.210508+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:40:14.190573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:40:37.923224+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:41:01.856527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.82:0/677169328":"2024-06-07T07:37:37.500868+0000","172.21.15.82:0/3317713296":"2024-06-07T07:37:37.500868+0000","172.21.15.82:6800/1192692524":"2024-06-07T07:35:54.660448+0000","172.21.15.82:6801/1662212559":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/2648665552":"2024-06-07T07:36:25.351026+0000","172.21.15.82:6801/1192692524":"2024-06-07T07:35:54.660448+0000","172.21.15.82:0/3728074668":"2024-06-07T07:35:54.660448+0000","172.21.15.82:0/1222848881":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/3401954453":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/2354085137":"2024-06-07T07:35:54.660448+0000","172.21.15.82:6800/493557010":"2024-06-07T07:37:37.500868+0000","172.21.15.82:6801/493557010":"2024-06-07T07:37:37.500868+0000","172.21.15.82:0/3313636867":"2024-06-07T07:35:54.660448+0000","172.21.15.82:6800/1662212559":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/1196491324":"2024-06-07T07:37:37.500868+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-06T07:41:09.787 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.785+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe4d006aef0 msgr2=0x7fe4d006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:09.787 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.785+0000 7fe4e93b1640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe4d006aef0 0x7fe4d006d3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc007dc0 tx=0x7fe4dc0072f0 comp rx=0 tx=0).stop 2024-06-06T07:41:09.787 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.785+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e4105240 msgr2=0x7fe4e4196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:09.787 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.785+0000 7fe4e93b1640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e4105240 0x7fe4e4196880 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc02a9c0 tx=0x7fe4cc0055a0 comp rx=0 tx=0).stop 2024-06-06T07:41:09.787 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.785+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 shutdown_connections 2024-06-06T07:41:09.787 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.785+0000 7fe4e93b1640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe4d006aef0 0x7fe4d006d3b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:09.787 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.785+0000 7fe4e93b1640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4e4105240 0x7fe4e4196880 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:09.787 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.785+0000 7fe4e93b1640 1 --2- 172.21.15.82:0/1237385250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4e40fe770 0x7fe4e4196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:09.787 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.785+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 >> 172.21.15.82:0/1237385250 conn(0x7fe4e4075570 msgr2=0x7fe4e40fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:09.788 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.786+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 shutdown_connections 2024-06-06T07:41:09.788 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:09.786+0000 7fe4e93b1640 1 -- 172.21.15.82:0/1237385250 wait complete. 2024-06-06T07:41:10.523 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-06T07:39:55.812493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-06T07:41:10.524 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-06T07:41:10.733 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:10.855 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:10 smithi082 ceph-mon[29354]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:10.855 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:10 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1237385250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T07:41:11.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:10 smithi184 ceph-mon[37288]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:11.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:10 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1237385250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T07:41:12.462 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.444+0000 7f41f38cf640 1 -- 172.21.15.82:0/3321110281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ffd50 msgr2=0x7f41ec103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:12.462 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.444+0000 7f41f38cf640 1 --2- 172.21.15.82:0/3321110281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ffd50 0x7f41ec103da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009880 tx=0x7f41dc02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:12.462 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.445+0000 7f41f38cf640 1 -- 172.21.15.82:0/3321110281 shutdown_connections 2024-06-06T07:41:12.462 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.445+0000 7f41f38cf640 1 --2- 172.21.15.82:0/3321110281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ffd50 0x7f41ec103da0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:12.462 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.445+0000 7f41f38cf640 1 --2- 172.21.15.82:0/3321110281 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41ec0ff3a0 0x7f41ec0ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.445+0000 7f41f38cf640 1 -- 172.21.15.82:0/3321110281 >> 172.21.15.82:0/3321110281 conn(0x7f41ec0fac50 msgr2=0x7f41ec0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.445+0000 7f41f38cf640 1 -- 172.21.15.82:0/3321110281 shutdown_connections 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.445+0000 7f41f38cf640 1 -- 172.21.15.82:0/3321110281 wait complete. 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.445+0000 7f41f38cf640 1 Processor -- start 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.445+0000 7f41f38cf640 1 -- start start 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.445+0000 7f41f38cf640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ff3a0 0x7f41ec1971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f38cf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41ec0ffd50 0x7f41ec197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f38cf640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec197c50 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f38cf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec197dc0 con 0x7f41ec0ffd50 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f166b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ff3a0 0x7f41ec1971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f166b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ff3a0 0x7f41ec1971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:37184/0 (socket says 172.21.15.82:37184) 2024-06-06T07:41:12.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f166b640 1 -- 172.21.15.82:0/3454407402 learned_addr learned my addr 172.21.15.82:0/3454407402 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f0e6a640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41ec0ffd50 0x7f41ec197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f166b640 1 -- 172.21.15.82:0/3454407402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41ec0ffd50 msgr2=0x7f41ec197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f166b640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41ec0ffd50 0x7f41ec197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f166b640 1 -- 172.21.15.82:0/3454407402 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc009560 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f166b640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ff3a0 0x7f41ec1971d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f41e800b420 tx=0x7f41e800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41e27fc640 1 -- 172.21.15.82:0/3454407402 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41e8011800 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41e27fc640 1 -- 172.21.15.82:0/3454407402 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41e8011e20 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec1a0240 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41e27fc640 1 -- 172.21.15.82:0/3454407402 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41e800f2e0 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.446+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec1a0650 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.447+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ec100880 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.448+0000 7f41e27fc640 1 -- 172.21.15.82:0/3454407402 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f41e8010380 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.448+0000 7f41e27fc640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f41d806aef0 0x7f41d806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:12.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.448+0000 7f41f0e6a640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f41d806aef0 0x7f41d806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:12.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.448+0000 7f41e27fc640 1 -- 172.21.15.82:0/3454407402 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f41e8089570 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.449+0000 7f41f0e6a640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f41d806aef0 0x7f41d806d3b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009230 tx=0x7f41dc005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:12.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.452+0000 7f41e27fc640 1 -- 172.21.15.82:0/3454407402 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f41e8054b50 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.626 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.626+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41ec068d40 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.626 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.626+0000 7f41e27fc640 1 -- 172.21.15.82:0/3454407402 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f41e8058170 con 0x7f41ec0ff3a0 2024-06-06T07:41:12.627 INFO:teuthology.orchestra.run.smithi082.stdout:pg_num: 1 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f41d806aef0 msgr2=0x7f41d806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f41d806aef0 0x7f41d806d3b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009230 tx=0x7f41dc005a20 comp rx=0 tx=0).stop 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ff3a0 msgr2=0x7f41ec1971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ff3a0 0x7f41ec1971d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f41e800b420 tx=0x7f41e800b8e0 comp rx=0 tx=0).stop 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 shutdown_connections 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f41d806aef0 0x7f41d806d3b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41ec0ffd50 0x7f41ec197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 --2- 172.21.15.82:0/3454407402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41ec0ff3a0 0x7f41ec1971d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 >> 172.21.15.82:0/3454407402 conn(0x7f41ec0fac50 msgr2=0x7f41ec0fc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 shutdown_connections 2024-06-06T07:41:12.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:12.627+0000 7f41f38cf640 1 -- 172.21.15.82:0/3454407402 wait complete. 2024-06-06T07:41:12.924 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:12 smithi082 ceph-mon[29354]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:12.924 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:12 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3454407402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-06T07:41:13.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:12 smithi184 ceph-mon[37288]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:13.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:12 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3454407402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-06T07:41:13.330 INFO:tasks.cephadm:Setting up client nodes... 2024-06-06T07:41:13.330 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-06T07:41:13.534 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:14.921 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:14 smithi082 ceph-mon[29354]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:15.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:14 smithi184 ceph-mon[37288]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:15.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.147+0000 7f8d8003c640 1 -- 172.21.15.82:0/2972084549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d78100380 msgr2=0x7f8d78100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:15.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.147+0000 7f8d8003c640 1 --2- 172.21.15.82:0/2972084549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d78100380 0x7f8d78100800 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009880 tx=0x7f8d6802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:15.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.147+0000 7f8d8003c640 1 -- 172.21.15.82:0/2972084549 shutdown_connections 2024-06-06T07:41:15.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.147+0000 7f8d8003c640 1 --2- 172.21.15.82:0/2972084549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d78100380 0x7f8d78100800 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:15.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.147+0000 7f8d8003c640 1 --2- 172.21.15.82:0/2972084549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d780ff180 0x7f8d780ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:15.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.147+0000 7f8d8003c640 1 -- 172.21.15.82:0/2972084549 >> 172.21.15.82:0/2972084549 conn(0x7f8d780fa720 msgr2=0x7f8d780fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:15.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.147+0000 7f8d8003c640 1 -- 172.21.15.82:0/2972084549 shutdown_connections 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.147+0000 7f8d8003c640 1 -- 172.21.15.82:0/2972084549 wait complete. 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d8003c640 1 Processor -- start 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d8003c640 1 -- start start 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d8003c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d780ff180 0x7f8d78195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d8003c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d78100380 0x7f8d78196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d8003c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d78196860 con 0x7f8d780ff180 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d8003c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d781969d0 con 0x7f8d78100380 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d7ddd8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d780ff180 0x7f8d78195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d7ddd8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d780ff180 0x7f8d78195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47702/0 (socket says 172.21.15.82:47702) 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d7ddd8640 1 -- 172.21.15.82:0/2085655565 learned_addr learned my addr 172.21.15.82:0/2085655565 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d7d5d7640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d78100380 0x7f8d78196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:15.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d7ddd8640 1 -- 172.21.15.82:0/2085655565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d78100380 msgr2=0x7f8d78196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:15.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d7ddd8640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d78100380 0x7f8d78196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:15.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.148+0000 7f8d7ddd8640 1 -- 172.21.15.82:0/2085655565 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d68009560 con 0x7f8d780ff180 2024-06-06T07:41:15.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.149+0000 7f8d7ddd8640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d780ff180 0x7f8d78195d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400d8d0 tx=0x7f8d7400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:15.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.149+0000 7f8d6effd640 1 -- 172.21.15.82:0/2085655565 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d74009a30 con 0x7f8d780ff180 2024-06-06T07:41:15.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.149+0000 7f8d6effd640 1 -- 172.21.15.82:0/2085655565 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d74009b90 con 0x7f8d780ff180 2024-06-06T07:41:15.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.149+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7819b410 con 0x7f8d780ff180 2024-06-06T07:41:15.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.149+0000 7f8d6effd640 1 -- 172.21.15.82:0/2085655565 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d7400f680 con 0x7f8d780ff180 2024-06-06T07:41:15.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.149+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7819b9e0 con 0x7f8d780ff180 2024-06-06T07:41:15.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.150+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7818fe10 con 0x7f8d780ff180 2024-06-06T07:41:15.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.150+0000 7f8d6effd640 1 -- 172.21.15.82:0/2085655565 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f8d74009d00 con 0x7f8d780ff180 2024-06-06T07:41:15.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.151+0000 7f8d6effd640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8d6406aef0 0x7f8d6406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:15.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.151+0000 7f8d6effd640 1 -- 172.21.15.82:0/2085655565 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8d740899a0 con 0x7f8d780ff180 2024-06-06T07:41:15.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.151+0000 7f8d7d5d7640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8d6406aef0 0x7f8d6406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:15.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.151+0000 7f8d7d5d7640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8d6406aef0 0x7f8d6406d3b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000c00 tx=0x7f8d680314e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:15.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.155+0000 7f8d6effd640 1 -- 172.21.15.82:0/2085655565 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8d74055000 con 0x7f8d780ff180 2024-06-06T07:41:15.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.392+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f8d78108e80 con 0x7f8d780ff180 2024-06-06T07:41:15.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.397+0000 7f8d6effd640 1 -- 172.21.15.82:0/2085655565 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f8d740170f0 con 0x7f8d780ff180 2024-06-06T07:41:15.398 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2024-06-06T07:41:15.398 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAbaGFmJ/p8FxAAdblqvY75fl+xJ00q1N5edg== 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.399+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8d6406aef0 msgr2=0x7f8d6406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.399+0000 7f8d8003c640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8d6406aef0 0x7f8d6406d3b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000c00 tx=0x7f8d680314e0 comp rx=0 tx=0).stop 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.399+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d780ff180 msgr2=0x7f8d78195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.399+0000 7f8d8003c640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d780ff180 0x7f8d78195d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400d8d0 tx=0x7f8d7400dd90 comp rx=0 tx=0).stop 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.399+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 shutdown_connections 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.399+0000 7f8d8003c640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f8d6406aef0 0x7f8d6406d3b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.399+0000 7f8d8003c640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d78100380 0x7f8d78196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.399+0000 7f8d8003c640 1 --2- 172.21.15.82:0/2085655565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d780ff180 0x7f8d78195d50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.399+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 >> 172.21.15.82:0/2085655565 conn(0x7f8d780fa720 msgr2=0x7f8d78103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:15.400 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.400+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 shutdown_connections 2024-06-06T07:41:15.401 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:15.400+0000 7f8d8003c640 1 -- 172.21.15.82:0/2085655565 wait complete. 2024-06-06T07:41:16.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:15 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2085655565' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T07:41:16.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:15 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2085655565' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T07:41:16.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:15 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2085655565' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T07:41:16.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:15 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2085655565' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T07:41:16.193 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-06-06T07:41:16.194 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-06T07:41:16.194 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-06T07:41:16.231 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-06T07:41:16.455 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi184/config 2024-06-06T07:41:16.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:16 smithi082 ceph-mon[29354]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:16.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:16 smithi184 ceph-mon[37288]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:17.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.882+0000 7ff0d6540640 1 -- 172.21.15.184:0/1555444970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 msgr2=0x7ff0d0100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:17.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.882+0000 7ff0d6540640 1 --2- 172.21.15.184:0/1555444970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 0x7ff0d0100800 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009880 tx=0x7ff0c402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:17.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.882+0000 7ff0d6540640 1 -- 172.21.15.184:0/1555444970 shutdown_connections 2024-06-06T07:41:17.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.882+0000 7ff0d6540640 1 --2- 172.21.15.184:0/1555444970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 0x7ff0d0100800 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:17.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.882+0000 7ff0d6540640 1 --2- 172.21.15.184:0/1555444970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0d00ff180 0x7ff0d00ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:17.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.882+0000 7ff0d6540640 1 -- 172.21.15.184:0/1555444970 >> 172.21.15.184:0/1555444970 conn(0x7ff0d00fa720 msgr2=0x7ff0d00fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:17.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.882+0000 7ff0d6540640 1 -- 172.21.15.184:0/1555444970 shutdown_connections 2024-06-06T07:41:17.884 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.882+0000 7ff0d6540640 1 -- 172.21.15.184:0/1555444970 wait complete. 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0d6540640 1 Processor -- start 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0d6540640 1 -- start start 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0d6540640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0d00ff180 0x7ff0d0069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0d6540640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 0x7ff0d019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0d6540640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d019afd0 con 0x7ff0d00ff180 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0d6540640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d019b140 con 0x7ff0d0100380 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0cf7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 0x7ff0d019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0cf7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 0x7ff0d019aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:37778/0 (socket says 172.21.15.184:37778) 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0cf7fe640 1 -- 172.21.15.184:0/1476567547 learned_addr learned my addr 172.21.15.184:0/1476567547 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0cffff640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0d00ff180 0x7ff0d0069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0cf7fe640 1 -- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0d00ff180 msgr2=0x7ff0d0069250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0cf7fe640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0d00ff180 0x7ff0d0069250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:17.885 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.883+0000 7ff0cf7fe640 1 -- 172.21.15.184:0/1476567547 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4009560 con 0x7ff0d0100380 2024-06-06T07:41:17.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.884+0000 7ff0cf7fe640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 0x7ff0d019aa90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4005610 tx=0x7ff0c4005640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:17.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.884+0000 7ff0cd7fa640 1 -- 172.21.15.184:0/1476567547 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0c4004ba0 con 0x7ff0d0100380 2024-06-06T07:41:17.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.884+0000 7ff0cd7fa640 1 -- 172.21.15.184:0/1476567547 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0c4004d00 con 0x7ff0d0100380 2024-06-06T07:41:17.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.884+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d019b3c0 con 0x7ff0d0100380 2024-06-06T07:41:17.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.884+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d019b930 con 0x7ff0d0100380 2024-06-06T07:41:17.886 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.884+0000 7ff0cd7fa640 1 -- 172.21.15.184:0/1476567547 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0c4003bd0 con 0x7ff0d0100380 2024-06-06T07:41:17.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.885+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d018ecf0 con 0x7ff0d0100380 2024-06-06T07:41:17.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.885+0000 7ff0cd7fa640 1 -- 172.21.15.184:0/1476567547 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7ff0c4031640 con 0x7ff0d0100380 2024-06-06T07:41:17.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.886+0000 7ff0cd7fa640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff0b806aef0 0x7ff0b806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:17.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.886+0000 7ff0cd7fa640 1 -- 172.21.15.184:0/1476567547 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff0c40aaec0 con 0x7ff0d0100380 2024-06-06T07:41:17.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.886+0000 7ff0cffff640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff0b806aef0 0x7ff0b806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:17.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.886+0000 7ff0cffff640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff0b806aef0 0x7ff0b806d3b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff0d01001e0 tx=0x7ff0c000a2d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:17.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:17.890+0000 7ff0cd7fa640 1 -- 172.21.15.184:0/1476567547 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff0c4076540 con 0x7ff0d0100380 2024-06-06T07:41:18.127 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.126+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ff0d0107d60 con 0x7ff0d0100380 2024-06-06T07:41:18.132 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.132+0000 7ff0cd7fa640 1 -- 172.21.15.184:0/1476567547 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7ff0c404a050 con 0x7ff0d0100380 2024-06-06T07:41:18.132 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2024-06-06T07:41:18.132 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQAeaGFmJieuBxAA1cehMUowTLWLNS/70C1TcQ== 2024-06-06T07:41:18.134 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff0b806aef0 msgr2=0x7ff0b806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff0b806aef0 0x7ff0b806d3b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff0d01001e0 tx=0x7ff0c000a2d0 comp rx=0 tx=0).stop 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 msgr2=0x7ff0d019aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 0x7ff0d019aa90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4005610 tx=0x7ff0c4005640 comp rx=0 tx=0).stop 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 shutdown_connections 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff0b806aef0 0x7ff0b806d3b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d0100380 0x7ff0d019aa90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 --2- 172.21.15.184:0/1476567547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0d00ff180 0x7ff0d0069250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 >> 172.21.15.184:0/1476567547 conn(0x7ff0d00fa720 msgr2=0x7ff0d01039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 shutdown_connections 2024-06-06T07:41:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:2024-06-06T07:41:18.134+0000 7ff0d6540640 1 -- 172.21.15.184:0/1476567547 wait complete. 2024-06-06T07:41:18.634 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-06-06T07:41:18.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-06T07:41:18.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-06T07:41:18.672 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-06T07:41:18.672 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-06T07:41:18.673 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-06-06T07:41:18.890 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:19.012 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:18 smithi082 ceph-mon[29354]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:19.013 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:18 smithi082 ceph-mon[29354]: from='client.? 172.21.15.184:0/1476567547' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T07:41:19.013 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:18 smithi082 ceph-mon[29354]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T07:41:19.013 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:18 smithi082 ceph-mon[29354]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T07:41:19.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:18 smithi184 ceph-mon[37288]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:18 smithi184 ceph-mon[37288]: from='client.? 172.21.15.184:0/1476567547' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T07:41:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:18 smithi184 ceph-mon[37288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T07:41:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:18 smithi184 ceph-mon[37288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T07:41:20.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 -- 172.21.15.82:0/3966292137 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ff360 msgr2=0x7f2af00ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:20.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 --2- 172.21.15.82:0/3966292137 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ff360 0x7f2af00ff740 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009880 tx=0x7f2ae002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:20.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 -- 172.21.15.82:0/3966292137 shutdown_connections 2024-06-06T07:41:20.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 --2- 172.21.15.82:0/3966292137 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af00ffd10 0x7f2af0103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:20.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 --2- 172.21.15.82:0/3966292137 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ff360 0x7f2af00ff740 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:20.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 -- 172.21.15.82:0/3966292137 >> 172.21.15.82:0/3966292137 conn(0x7f2af0075e10 msgr2=0x7f2af00fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:20.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 -- 172.21.15.82:0/3966292137 shutdown_connections 2024-06-06T07:41:20.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 -- 172.21.15.82:0/3966292137 wait complete. 2024-06-06T07:41:20.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 Processor -- start 2024-06-06T07:41:20.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.644+0000 7f2af5d06640 1 -- start start 2024-06-06T07:41:20.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2af5d06640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af00ff360 0x7f2af0197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:20.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2af5d06640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ffd10 0x7f2af0197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:20.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2af5d06640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af0197ad0 con 0x7f2af00ffd10 2024-06-06T07:41:20.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2af5d06640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af0197c40 con 0x7f2af00ff360 2024-06-06T07:41:20.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2ae6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ffd10 0x7f2af0197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:20.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2ae6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ffd10 0x7f2af0197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47730/0 (socket says 172.21.15.82:47730) 2024-06-06T07:41:20.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2ae6ffd640 1 -- 172.21.15.82:0/2308165273 learned_addr learned my addr 172.21.15.82:0/2308165273 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:20.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2aef7fe640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af00ff360 0x7f2af0197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:20.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2ae6ffd640 1 -- 172.21.15.82:0/2308165273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af00ff360 msgr2=0x7f2af0197050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:20.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2ae6ffd640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af00ff360 0x7f2af0197050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:20.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2ae6ffd640 1 -- 172.21.15.82:0/2308165273 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae0009560 con 0x7f2af00ffd10 2024-06-06T07:41:20.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.645+0000 7f2ae6ffd640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ffd10 0x7f2af0197590 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800c8f0 tx=0x7f2ad800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:20.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.646+0000 7f2aed7fa640 1 -- 172.21.15.82:0/2308165273 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ad8004410 con 0x7f2af00ffd10 2024-06-06T07:41:20.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.646+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af01a00c0 con 0x7f2af00ffd10 2024-06-06T07:41:20.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.646+0000 7f2aed7fa640 1 -- 172.21.15.82:0/2308165273 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ad8007c60 con 0x7f2af00ffd10 2024-06-06T07:41:20.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.646+0000 7f2aed7fa640 1 -- 172.21.15.82:0/2308165273 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ad800f450 con 0x7f2af00ffd10 2024-06-06T07:41:20.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.646+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af01a0470 con 0x7f2af00ffd10 2024-06-06T07:41:20.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.647+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af0100840 con 0x7f2af00ffd10 2024-06-06T07:41:20.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.647+0000 7f2aed7fa640 1 -- 172.21.15.82:0/2308165273 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f2ad80077a0 con 0x7f2af00ffd10 2024-06-06T07:41:20.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.647+0000 7f2aed7fa640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2adc06aef0 0x7f2adc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:20.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.648+0000 7f2aef7fe640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2adc06aef0 0x7f2adc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:20.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.648+0000 7f2aed7fa640 1 -- 172.21.15.82:0/2308165273 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2ad8089b60 con 0x7f2af00ffd10 2024-06-06T07:41:20.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.648+0000 7f2aef7fe640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2adc06aef0 0x7f2adc06d3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0000c00 tx=0x7f2ae0005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:20.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.652+0000 7f2aed7fa640 1 -- 172.21.15.82:0/2308165273 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2ad8055140 con 0x7f2af00ffd10 2024-06-06T07:41:20.878 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.878+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f2af004e210 con 0x7f2af00ffd10 2024-06-06T07:41:20.881 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.881+0000 7f2aed7fa640 1 -- 172.21.15.82:0/2308165273 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+171352 (secure 0 0 0) 0x7f2ad8058760 con 0x7f2af00ffd10 2024-06-06T07:41:20.884 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2adc06aef0 msgr2=0x7f2adc06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2adc06aef0 0x7f2adc06d3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0000c00 tx=0x7f2ae0005a20 comp rx=0 tx=0).stop 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ffd10 msgr2=0x7f2af0197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ffd10 0x7f2af0197590 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800c8f0 tx=0x7f2ad800cdb0 comp rx=0 tx=0).stop 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 shutdown_connections 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2adc06aef0 0x7f2adc06d3b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2af00ffd10 0x7f2af0197590 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 --2- 172.21.15.82:0/2308165273 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af00ff360 0x7f2af0197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 >> 172.21.15.82:0/2308165273 conn(0x7f2af0075e10 msgr2=0x7f2af0102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 shutdown_connections 2024-06-06T07:41:20.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:20.884+0000 7f2af5d06640 1 -- 172.21.15.82:0/2308165273 wait complete. 2024-06-06T07:41:21.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:20 smithi082 ceph-mon[29354]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:21.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:20 smithi184 ceph-mon[37288]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:21.647 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":19,"flags":0,"active_gid":14221,"active_name":"smithi082.kocfig","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":2676832140},{"type":"v1","addr":"172.21.15.82:6801","nonce":2676832140}]},"active_addr":"172.21.15.82:6801/2676832140","active_change":"2024-06-06T07:37:37.500976+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14246,"name":"smithi184.cqttux","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.82:8443/","prometheus":"http://172.21.15.82:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":1312319606}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":1671958896}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":807186706}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":3251595324}]}]} 2024-06-06T07:41:21.651 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-06T07:41:21.651 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-06T07:41:21.651 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-06T07:41:21.855 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:21.977 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:21 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2308165273' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-06T07:41:22.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:21 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2308165273' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-06T07:41:22.990 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:22 smithi082 ceph-mon[29354]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:22.990 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:41:23.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:22 smithi184 ceph-mon[37288]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:41:23.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.497+0000 7f13a397b640 1 -- 172.21.15.82:0/2382198919 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 msgr2=0x7f139c103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:23.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.497+0000 7f13a397b640 1 --2- 172.21.15.82:0/2382198919 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 0x7f139c103ff0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f138c009880 tx=0x7f138c02a530 comp rx=0 tx=0).stop 2024-06-06T07:41:23.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.497+0000 7f13a397b640 1 -- 172.21.15.82:0/2382198919 shutdown_connections 2024-06-06T07:41:23.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.497+0000 7f13a397b640 1 --2- 172.21.15.82:0/2382198919 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 0x7f139c103ff0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:23.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.497+0000 7f13a397b640 1 --2- 172.21.15.82:0/2382198919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f139c0ff200 0x7f139c0ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:23.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.497+0000 7f13a397b640 1 -- 172.21.15.82:0/2382198919 >> 172.21.15.82:0/2382198919 conn(0x7f139c0fac80 msgr2=0x7f139c0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.498+0000 7f13a397b640 1 -- 172.21.15.82:0/2382198919 shutdown_connections 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.498+0000 7f13a397b640 1 -- 172.21.15.82:0/2382198919 wait complete. 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.498+0000 7f13a397b640 1 Processor -- start 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.498+0000 7f13a397b640 1 -- start start 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a397b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f139c0ff200 0x7f139c1a2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a397b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 0x7f139c1a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a397b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c19c7a0 con 0x7f139c0ffb20 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a397b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c19c910 con 0x7f139c0ff200 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a0f16640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 0x7f139c1a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a0f16640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 0x7f139c1a2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47754/0 (socket says 172.21.15.82:47754) 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a0f16640 1 -- 172.21.15.82:0/1307068124 learned_addr learned my addr 172.21.15.82:0/1307068124 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a1717640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f139c0ff200 0x7f139c1a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a0f16640 1 -- 172.21.15.82:0/1307068124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f139c0ff200 msgr2=0x7f139c1a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:23.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a0f16640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f139c0ff200 0x7f139c1a2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:23.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a0f16640 1 -- 172.21.15.82:0/1307068124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c009560 con 0x7f139c0ffb20 2024-06-06T07:41:23.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a0f16640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 0x7f139c1a2ba0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f138c02aa50 tx=0x7f138c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:23.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13927fc640 1 -- 172.21.15.82:0/1307068124 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f138c0048d0 con 0x7f139c0ffb20 2024-06-06T07:41:23.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c19cb90 con 0x7f139c0ffb20 2024-06-06T07:41:23.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13927fc640 1 -- 172.21.15.82:0/1307068124 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f138c031640 con 0x7f139c0ffb20 2024-06-06T07:41:23.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13927fc640 1 -- 172.21.15.82:0/1307068124 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f138c003820 con 0x7f139c0ffb20 2024-06-06T07:41:23.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.499+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139c19d080 con 0x7f139c0ffb20 2024-06-06T07:41:23.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.500+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139c069060 con 0x7f139c0ffb20 2024-06-06T07:41:23.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.501+0000 7f13927fc640 1 -- 172.21.15.82:0/1307068124 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f138c02ad50 con 0x7f139c0ffb20 2024-06-06T07:41:23.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.501+0000 7f13927fc640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f138806aef0 0x7f138806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:23.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.502+0000 7f13a1717640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f138806aef0 0x7f138806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:23.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.502+0000 7f13927fc640 1 -- 172.21.15.82:0/1307068124 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f138c0aa890 con 0x7f139c0ffb20 2024-06-06T07:41:23.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.502+0000 7f13a1717640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f138806aef0 0x7f138806d3b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1398009990 tx=0x7f1398009110 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:23.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.507+0000 7f13927fc640 1 -- 172.21.15.82:0/1307068124 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f138c075ef0 con 0x7f139c0ffb20 2024-06-06T07:41:23.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.680+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f139c19dc60 con 0x7f139c0ffb20 2024-06-06T07:41:23.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.681+0000 7f13927fc640 1 -- 172.21.15.82:0/1307068124 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11336 (secure 0 0 0) 0x7f138c079510 con 0x7f139c0ffb20 2024-06-06T07:41:23.682 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:41:23.682 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":40,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","created":"2024-06-06T07:35:28.756873+0000","modified":"2024-06-06T07:41:05.704570+0000","last_up_change":"2024-06-06T07:41:04.555363+0000","last_in_change":"2024-06-06T07:40:45.206953+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-06T07:39:55.812493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"919724ed-ba2c-4df8-aa46-cd599dcf0647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6803","nonce":879999521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6805","nonce":879999521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6809","nonce":879999521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6807","nonce":879999521}]},"public_addr":"172.21.15.82:6803/879999521","cluster_addr":"172.21.15.82:6805/879999521","heartbeat_back_addr":"172.21.15.82:6809/879999521","heartbeat_front_addr":"172.21.15.82:6807/879999521","state":["exists","up"]},{"osd":1,"uuid":"6d45e246-1b80-49e1-b33a-a8021b1e5013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6811","nonce":1482740538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6813","nonce":1482740538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6817","nonce":1482740538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6815","nonce":1482740538}]},"public_addr":"172.21.15.82:6811/1482740538","cluster_addr":"172.21.15.82:6813/1482740538","heartbeat_back_addr":"172.21.15.82:6817/1482740538","heartbeat_front_addr":"172.21.15.82:6815/1482740538","state":["exists","up"]},{"osd":2,"uuid":"b83ec350-58d2-4ebc-bc70-29823dfa9320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6819","nonce":1581462724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6821","nonce":1581462724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6825","nonce":1581462724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6823","nonce":1581462724}]},"public_addr":"172.21.15.82:6819/1581462724","cluster_addr":"172.21.15.82:6821/1581462724","heartbeat_back_addr":"172.21.15.82:6825/1581462724","heartbeat_front_addr":"172.21.15.82:6823/1581462724","state":["exists","up"]},{"osd":3,"uuid":"4b7500c1-d624-43fe-8998-c20f90fd7406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6801","nonce":3221286727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6803","nonce":3221286727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6807","nonce":3221286727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6805","nonce":3221286727}]},"public_addr":"172.21.15.184:6801/3221286727","cluster_addr":"172.21.15.184:6803/3221286727","heartbeat_back_addr":"172.21.15.184:6807/3221286727","heartbeat_front_addr":"172.21.15.184:6805/3221286727","state":["exists","up"]},{"osd":4,"uuid":"f84b5bb0-f47c-44c4-a092-8d5580f52b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6809","nonce":2263803443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6811","nonce":2263803443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6815","nonce":2263803443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6813","nonce":2263803443}]},"public_addr":"172.21.15.184:6809/2263803443","cluster_addr":"172.21.15.184:6811/2263803443","heartbeat_back_addr":"172.21.15.184:6815/2263803443","heartbeat_front_addr":"172.21.15.184:6813/2263803443","state":["exists","up"]},{"osd":5,"uuid":"4c6802b9-a205-4b2c-bd29-db1d2e8e80a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6817","nonce":2823055081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6819","nonce":2823055081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6823","nonce":2823055081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6821","nonce":2823055081}]},"public_addr":"172.21.15.184:6817/2823055081","cluster_addr":"172.21.15.184:6819/2823055081","heartbeat_back_addr":"172.21.15.184:6823/2823055081","heartbeat_front_addr":"172.21.15.184:6821/2823055081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:38:55.385571+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:39:23.464338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:39:51.210508+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:40:14.190573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:40:37.923224+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:41:01.856527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.82:0/677169328":"2024-06-07T07:37:37.500868+0000","172.21.15.82:0/3317713296":"2024-06-07T07:37:37.500868+0000","172.21.15.82:6800/1192692524":"2024-06-07T07:35:54.660448+0000","172.21.15.82:6801/1662212559":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/2648665552":"2024-06-07T07:36:25.351026+0000","172.21.15.82:6801/1192692524":"2024-06-07T07:35:54.660448+0000","172.21.15.82:0/3728074668":"2024-06-07T07:35:54.660448+0000","172.21.15.82:0/1222848881":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/3401954453":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/2354085137":"2024-06-07T07:35:54.660448+0000","172.21.15.82:6800/493557010":"2024-06-07T07:37:37.500868+0000","172.21.15.82:6801/493557010":"2024-06-07T07:37:37.500868+0000","172.21.15.82:0/3313636867":"2024-06-07T07:35:54.660448+0000","172.21.15.82:6800/1662212559":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/1196491324":"2024-06-07T07:37:37.500868+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-06T07:41:23.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f138806aef0 msgr2=0x7f138806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:23.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f138806aef0 0x7f138806d3b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1398009990 tx=0x7f1398009110 comp rx=0 tx=0).stop 2024-06-06T07:41:23.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 msgr2=0x7f139c1a2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:23.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 0x7f139c1a2ba0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f138c02aa50 tx=0x7f138c0055a0 comp rx=0 tx=0).stop 2024-06-06T07:41:23.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 shutdown_connections 2024-06-06T07:41:23.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f138806aef0 0x7f138806d3b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:23.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f139c0ffb20 0x7f139c1a2ba0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:23.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 --2- 172.21.15.82:0/1307068124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f139c0ff200 0x7f139c1a2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:23.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 >> 172.21.15.82:0/1307068124 conn(0x7f139c0fac80 msgr2=0x7f139c10ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:23.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.683+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 shutdown_connections 2024-06-06T07:41:23.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:23.684+0000 7f13a397b640 1 -- 172.21.15.82:0/1307068124 wait complete. 2024-06-06T07:41:24.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:23 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1307068124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T07:41:24.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:23 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1307068124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T07:41:24.319 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-06T07:41:24.320 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-06T07:41:24.524 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:24.958 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:24 smithi082 ceph-mon[29354]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:25.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:24 smithi184 ceph-mon[37288]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:25.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.962+0000 7f517ae6a640 1 -- 172.21.15.82:0/4055059119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100e20 msgr2=0x7f5174104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:25.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.962+0000 7f517ae6a640 1 --2- 172.21.15.82:0/4055059119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100e20 0x7f5174104e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5160009880 tx=0x7f516002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:25.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.963+0000 7f517ae6a640 1 -- 172.21.15.82:0/4055059119 shutdown_connections 2024-06-06T07:41:25.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.963+0000 7f517ae6a640 1 --2- 172.21.15.82:0/4055059119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100e20 0x7f5174104e70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:25.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.963+0000 7f517ae6a640 1 --2- 172.21.15.82:0/4055059119 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5174100470 0x7f5174100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.963+0000 7f517ae6a640 1 -- 172.21.15.82:0/4055059119 >> 172.21.15.82:0/4055059119 conn(0x7f51740fbd20 msgr2=0x7f51740fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.963+0000 7f517ae6a640 1 -- 172.21.15.82:0/4055059119 shutdown_connections 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.963+0000 7f517ae6a640 1 -- 172.21.15.82:0/4055059119 wait complete. 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f517ae6a640 1 Processor -- start 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f517ae6a640 1 -- start start 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f517ae6a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100470 0x7f5174196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f517ae6a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5174100e20 0x7f5174196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f517ae6a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5174196ee0 con 0x7f5174100470 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f517ae6a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517419ac00 con 0x7f5174100e20 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f5178c06640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100470 0x7f5174196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f5178c06640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100470 0x7f5174196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42460/0 (socket says 172.21.15.82:42460) 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f5178c06640 1 -- 172.21.15.82:0/235109302 learned_addr learned my addr 172.21.15.82:0/235109302 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f5173fff640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5174100e20 0x7f5174196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:25.967 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f5178c06640 1 -- 172.21.15.82:0/235109302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5174100e20 msgr2=0x7f5174196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:25.967 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f5178c06640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5174100e20 0x7f5174196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:25.967 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.964+0000 7f5178c06640 1 -- 172.21.15.82:0/235109302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5160009560 con 0x7f5174100470 2024-06-06T07:41:25.967 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.965+0000 7f5178c06640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100470 0x7f5174196310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f516800b420 tx=0x7f516800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:25.967 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.965+0000 7f5171ffb640 1 -- 172.21.15.82:0/235109302 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5168011800 con 0x7f5174100470 2024-06-06T07:41:25.967 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.965+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517419aee0 con 0x7f5174100470 2024-06-06T07:41:25.967 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.965+0000 7f5171ffb640 1 -- 172.21.15.82:0/235109302 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5168011e20 con 0x7f5174100470 2024-06-06T07:41:25.967 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.965+0000 7f5171ffb640 1 -- 172.21.15.82:0/235109302 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f516800f2e0 con 0x7f5174100470 2024-06-06T07:41:25.967 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.965+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517419b400 con 0x7f5174100470 2024-06-06T07:41:25.968 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.966+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5174101950 con 0x7f5174100470 2024-06-06T07:41:25.968 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.966+0000 7f5171ffb640 1 -- 172.21.15.82:0/235109302 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f516800f5c0 con 0x7f5174100470 2024-06-06T07:41:25.968 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.967+0000 7f5171ffb640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f516406af00 0x7f516406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:25.968 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.967+0000 7f5171ffb640 1 -- 172.21.15.82:0/235109302 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f51680895a0 con 0x7f5174100470 2024-06-06T07:41:25.968 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.967+0000 7f5173fff640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f516406af00 0x7f516406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:25.969 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.968+0000 7f5173fff640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f516406af00 0x7f516406d3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5160009230 tx=0x7f5160004280 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:25.971 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:25.971+0000 7f5171ffb640 1 -- 172.21.15.82:0/235109302 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5168054d40 con 0x7f5174100470 2024-06-06T07:41:26.145 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.145+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f517404e210 con 0x7f5174100470 2024-06-06T07:41:26.146 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.146+0000 7f5171ffb640 1 -- 172.21.15.82:0/235109302 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11336 (secure 0 0 0) 0x7f5168058360 con 0x7f5174100470 2024-06-06T07:41:26.146 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:41:26.146 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":40,"fsid":"355cdb60-23d7-11ef-bc9b-c7b262605968","created":"2024-06-06T07:35:28.756873+0000","modified":"2024-06-06T07:41:05.704570+0000","last_up_change":"2024-06-06T07:41:04.555363+0000","last_in_change":"2024-06-06T07:40:45.206953+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-06T07:39:55.812493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"919724ed-ba2c-4df8-aa46-cd599dcf0647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6803","nonce":879999521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6805","nonce":879999521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6809","nonce":879999521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":879999521},{"type":"v1","addr":"172.21.15.82:6807","nonce":879999521}]},"public_addr":"172.21.15.82:6803/879999521","cluster_addr":"172.21.15.82:6805/879999521","heartbeat_back_addr":"172.21.15.82:6809/879999521","heartbeat_front_addr":"172.21.15.82:6807/879999521","state":["exists","up"]},{"osd":1,"uuid":"6d45e246-1b80-49e1-b33a-a8021b1e5013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6811","nonce":1482740538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6813","nonce":1482740538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6817","nonce":1482740538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1482740538},{"type":"v1","addr":"172.21.15.82:6815","nonce":1482740538}]},"public_addr":"172.21.15.82:6811/1482740538","cluster_addr":"172.21.15.82:6813/1482740538","heartbeat_back_addr":"172.21.15.82:6817/1482740538","heartbeat_front_addr":"172.21.15.82:6815/1482740538","state":["exists","up"]},{"osd":2,"uuid":"b83ec350-58d2-4ebc-bc70-29823dfa9320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6819","nonce":1581462724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6821","nonce":1581462724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6825","nonce":1581462724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1581462724},{"type":"v1","addr":"172.21.15.82:6823","nonce":1581462724}]},"public_addr":"172.21.15.82:6819/1581462724","cluster_addr":"172.21.15.82:6821/1581462724","heartbeat_back_addr":"172.21.15.82:6825/1581462724","heartbeat_front_addr":"172.21.15.82:6823/1581462724","state":["exists","up"]},{"osd":3,"uuid":"4b7500c1-d624-43fe-8998-c20f90fd7406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6801","nonce":3221286727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6803","nonce":3221286727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6807","nonce":3221286727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3221286727},{"type":"v1","addr":"172.21.15.184:6805","nonce":3221286727}]},"public_addr":"172.21.15.184:6801/3221286727","cluster_addr":"172.21.15.184:6803/3221286727","heartbeat_back_addr":"172.21.15.184:6807/3221286727","heartbeat_front_addr":"172.21.15.184:6805/3221286727","state":["exists","up"]},{"osd":4,"uuid":"f84b5bb0-f47c-44c4-a092-8d5580f52b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6809","nonce":2263803443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6811","nonce":2263803443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6815","nonce":2263803443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2263803443},{"type":"v1","addr":"172.21.15.184:6813","nonce":2263803443}]},"public_addr":"172.21.15.184:6809/2263803443","cluster_addr":"172.21.15.184:6811/2263803443","heartbeat_back_addr":"172.21.15.184:6815/2263803443","heartbeat_front_addr":"172.21.15.184:6813/2263803443","state":["exists","up"]},{"osd":5,"uuid":"4c6802b9-a205-4b2c-bd29-db1d2e8e80a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6817","nonce":2823055081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6819","nonce":2823055081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6823","nonce":2823055081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":2823055081},{"type":"v1","addr":"172.21.15.184:6821","nonce":2823055081}]},"public_addr":"172.21.15.184:6817/2823055081","cluster_addr":"172.21.15.184:6819/2823055081","heartbeat_back_addr":"172.21.15.184:6823/2823055081","heartbeat_front_addr":"172.21.15.184:6821/2823055081","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:38:55.385571+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:39:23.464338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:39:51.210508+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:40:14.190573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:40:37.923224+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T07:41:01.856527+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.82:0/677169328":"2024-06-07T07:37:37.500868+0000","172.21.15.82:0/3317713296":"2024-06-07T07:37:37.500868+0000","172.21.15.82:6800/1192692524":"2024-06-07T07:35:54.660448+0000","172.21.15.82:6801/1662212559":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/2648665552":"2024-06-07T07:36:25.351026+0000","172.21.15.82:6801/1192692524":"2024-06-07T07:35:54.660448+0000","172.21.15.82:0/3728074668":"2024-06-07T07:35:54.660448+0000","172.21.15.82:0/1222848881":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/3401954453":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/2354085137":"2024-06-07T07:35:54.660448+0000","172.21.15.82:6800/493557010":"2024-06-07T07:37:37.500868+0000","172.21.15.82:6801/493557010":"2024-06-07T07:37:37.500868+0000","172.21.15.82:0/3313636867":"2024-06-07T07:35:54.660448+0000","172.21.15.82:6800/1662212559":"2024-06-07T07:36:25.351026+0000","172.21.15.82:0/1196491324":"2024-06-07T07:37:37.500868+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-06T07:41:26.148 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f516406af00 msgr2=0x7f516406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:26.148 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f516406af00 0x7f516406d3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5160009230 tx=0x7f5160004280 comp rx=0 tx=0).stop 2024-06-06T07:41:26.148 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100470 msgr2=0x7f5174196310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:26.148 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100470 0x7f5174196310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f516800b420 tx=0x7f516800b8e0 comp rx=0 tx=0).stop 2024-06-06T07:41:26.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 shutdown_connections 2024-06-06T07:41:26.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f516406af00 0x7f516406d3c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:26.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5174100e20 0x7f5174196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:26.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 --2- 172.21.15.82:0/235109302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5174100470 0x7f5174196310 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:26.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 >> 172.21.15.82:0/235109302 conn(0x7f51740fbd20 msgr2=0x7f51740fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:26.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 shutdown_connections 2024-06-06T07:41:26.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:26.148+0000 7f517ae6a640 1 -- 172.21.15.82:0/235109302 wait complete. 2024-06-06T07:41:26.926 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:26 smithi082 ceph-mon[29354]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:26.927 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:26 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/235109302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T07:41:26.949 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-06T07:41:26.949 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-06T07:41:26.950 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-06T07:41:26.951 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-06T07:41:26.951 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-06T07:41:26.952 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-06T07:41:27.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:26 smithi184 ceph-mon[37288]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:27.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:26 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/235109302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T07:41:27.236 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:27.236 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:28.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:28 smithi082 ceph-mon[29354]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:29.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:28 smithi184 ceph-mon[37288]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:30.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.228+0000 7f62e650c640 1 -- 172.21.15.82:0/2095391911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 msgr2=0x7f62e010a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:30.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.228+0000 7f62e650c640 1 --2- 172.21.15.82:0/2095391911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 0x7f62e010a220 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009880 tx=0x7f62d402a530 comp rx=0 tx=0).stop 2024-06-06T07:41:30.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.228+0000 7f62e650c640 1 -- 172.21.15.82:0/2095391911 shutdown_connections 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.228+0000 7f62e650c640 1 --2- 172.21.15.82:0/2095391911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 0x7f62e010a220 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.228+0000 7f62e650c640 1 --2- 172.21.15.82:0/2095391911 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62e0101410 0x7f62e01017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.228+0000 7f62e650c640 1 -- 172.21.15.82:0/2095391911 >> 172.21.15.82:0/2095391911 conn(0x7f62e00fac80 msgr2=0x7f62e00fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.228+0000 7f62e650c640 1 -- 172.21.15.82:0/2095391911 shutdown_connections 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.228+0000 7f62e650c640 1 -- 172.21.15.82:0/2095391911 wait complete. 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62e650c640 1 Processor -- start 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62e650c640 1 -- start start 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62e650c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62e0101410 0x7f62e0196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62e650c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 0x7f62e0196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62e650c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e0196e20 con 0x7f62e0101d30 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62e650c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e019ab40 con 0x7f62e0101410 2024-06-06T07:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62df7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 0x7f62e0196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62df7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 0x7f62e0196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42470/0 (socket says 172.21.15.82:42470) 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62df7fe640 1 -- 172.21.15.82:0/3751190372 learned_addr learned my addr 172.21.15.82:0/3751190372 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.229+0000 7f62dffff640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62e0101410 0x7f62e0196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.230+0000 7f62df7fe640 1 -- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62e0101410 msgr2=0x7f62e0196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.230+0000 7f62df7fe640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62e0101410 0x7f62e0196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.230+0000 7f62df7fe640 1 -- 172.21.15.82:0/3751190372 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4009560 con 0x7f62e0101d30 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.230+0000 7f62df7fe640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 0x7f62e0196790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f62d402aa50 tx=0x7f62d40055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.230+0000 7f62dd7fa640 1 -- 172.21.15.82:0/3751190372 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62d40048d0 con 0x7f62e0101d30 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.230+0000 7f62dd7fa640 1 -- 172.21.15.82:0/3751190372 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62d4031640 con 0x7f62e0101d30 2024-06-06T07:41:30.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.230+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e019adc0 con 0x7f62e0101d30 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.230+0000 7f62dd7fa640 1 -- 172.21.15.82:0/3751190372 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62d4003820 con 0x7f62e0101d30 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.230+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e019b2b0 con 0x7f62e0101d30 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.231+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62e010d800 con 0x7f62e0101d30 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.232+0000 7f62dd7fa640 1 -- 172.21.15.82:0/3751190372 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f62d402ad50 con 0x7f62e0101d30 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.232+0000 7f62dd7fa640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f62c806aef0 0x7f62c806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.232+0000 7f62dd7fa640 1 -- 172.21.15.82:0/3751190372 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f62d40aa8b0 con 0x7f62e0101d30 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.232+0000 7f62dffff640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f62c806aef0 0x7f62c806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.232+0000 7f62dd7fa640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] conn(0x7f62c8070c80 0x7f62c80730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.232+0000 7f62dd7fa640 1 -- 172.21.15.82:0/3751190372 --> [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62c8073750 con 0x7f62c8070c80 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.232+0000 7f62dd7fa640 1 -- 172.21.15.82:0/3751190372 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62d40af2a0 con 0x7f62e0101d30 2024-06-06T07:41:30.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.232+0000 7f62dffff640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f62c806aef0 0x7f62c806d3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f62d0005fd0 tx=0x7f62d000b390 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:30.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.233+0000 7f62e4aa9640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] conn(0x7f62c8070c80 0x7f62c80730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:30.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.233+0000 7f62e4aa9640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] conn(0x7f62c8070c80 0x7f62c80730a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:30.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.234+0000 7f62dd7fa640 1 -- 172.21.15.82:0/3751190372 <== osd.4 v2:172.21.15.184:6808/2263803443 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f62c8073750 con 0x7f62c8070c80 2024-06-06T07:41:30.247 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.247+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 --> [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f62e019b710 con 0x7f62c8070c80 2024-06-06T07:41:30.248 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62dd7fa640 1 -- 172.21.15.82:0/3751190372 <== osd.4 v2:172.21.15.184:6808/2263803443 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f62e019b710 con 0x7f62c8070c80 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] conn(0x7f62c8070c80 msgr2=0x7f62c80730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] conn(0x7f62c8070c80 0x7f62c80730a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f62c806aef0 msgr2=0x7f62c806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f62c806aef0 0x7f62c806d3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f62d0005fd0 tx=0x7f62d000b390 comp rx=0 tx=0).stop 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 msgr2=0x7f62e0196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 0x7f62e0196790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f62d402aa50 tx=0x7f62d40055a0 comp rx=0 tx=0).stop 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 shutdown_connections 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f62c806aef0 0x7f62c806d3b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:6808/2263803443,v1:172.21.15.184:6809/2263803443] conn(0x7f62c8070c80 0x7f62c80730a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f62e0101d30 0x7f62e0196790 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 --2- 172.21.15.82:0/3751190372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62e0101410 0x7f62e0196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 >> 172.21.15.82:0/3751190372 conn(0x7f62e00fac80 msgr2=0x7f62e01026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:30.249 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 shutdown_connections 2024-06-06T07:41:30.250 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:30.248+0000 7f62e650c640 1 -- 172.21.15.82:0/3751190372 wait complete. 2024-06-06T07:41:30.876 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:30 smithi082 ceph-mon[29354]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:31.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:30 smithi184 ceph-mon[37288]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:32.581 INFO:teuthology.orchestra.run.smithi082.stdout:146028888077 2024-06-06T07:41:32.582 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-06T07:41:32.816 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:32 smithi082 ceph-mon[29354]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:33.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:32 smithi184 ceph-mon[37288]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:34.963 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:34 smithi082 ceph-mon[29354]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:35.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:34 smithi184 ceph-mon[37288]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:36.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.412+0000 7f40008f8640 1 -- 172.21.15.82:0/994168131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 msgr2=0x7f3ffc0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:36.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.412+0000 7f40008f8640 1 --2- 172.21.15.82:0/994168131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 0x7f3ffc0ff790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3fec009880 tx=0x7f3fec02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.412+0000 7f40008f8640 1 -- 172.21.15.82:0/994168131 shutdown_connections 2024-06-06T07:41:36.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.412+0000 7f40008f8640 1 --2- 172.21.15.82:0/994168131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc0ffd60 0x7f3ffc103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.412+0000 7f40008f8640 1 --2- 172.21.15.82:0/994168131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 0x7f3ffc0ff790 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.412+0000 7f40008f8640 1 -- 172.21.15.82:0/994168131 >> 172.21.15.82:0/994168131 conn(0x7f3ffc0fac80 msgr2=0x7f3ffc0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:36.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.412+0000 7f40008f8640 1 -- 172.21.15.82:0/994168131 shutdown_connections 2024-06-06T07:41:36.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.412+0000 7f40008f8640 1 -- 172.21.15.82:0/994168131 wait complete. 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f40008f8640 1 Processor -- start 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f40008f8640 1 -- start start 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f40008f8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 0x7f3ffc193f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f40008f8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc0ffd60 0x7f3ffc194470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f40008f8640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc194a40 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f40008f8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc194bb0 con 0x7f3ffc0ffd60 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f3ffa59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 0x7f3ffc193f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f3ffa59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 0x7f3ffc193f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:58258/0 (socket says 172.21.15.82:58258) 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f3ffa59c640 1 -- 172.21.15.82:0/776990707 learned_addr learned my addr 172.21.15.82:0/776990707 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f3ff9d9b640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc0ffd60 0x7f3ffc194470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:36.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f3ffa59c640 1 -- 172.21.15.82:0/776990707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc0ffd60 msgr2=0x7f3ffc194470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:36.417 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f3ffa59c640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc0ffd60 0x7f3ffc194470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.417 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.413+0000 7f3ffa59c640 1 -- 172.21.15.82:0/776990707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec009560 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.417 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.414+0000 7f3ffa59c640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 0x7f3ffc193f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3fec02a9c0 tx=0x7f3fec0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:36.417 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.414+0000 7f3ff37fe640 1 -- 172.21.15.82:0/776990707 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fec004aa0 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.417 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.414+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ffc1a66f0 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.417 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.414+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ffc1a6aa0 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.417 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.414+0000 7f3ff37fe640 1 -- 172.21.15.82:0/776990707 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fec004c00 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.417 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.414+0000 7f3ff37fe640 1 -- 172.21.15.82:0/776990707 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fec003980 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.417 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.415+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3fdc000fc0 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.416+0000 7f3ff37fe640 1 -- 172.21.15.82:0/776990707 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f3fec003ae0 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.416+0000 7f3ff37fe640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f3fe806aef0 0x7f3fe806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:36.418 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.416+0000 7f3ff37fe640 1 -- 172.21.15.82:0/776990707 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3fec0ab070 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.419 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.416+0000 7f3ff9d9b640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f3fe806aef0 0x7f3fe806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:36.419 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.416+0000 7f3ff37fe640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] conn(0x7f3fe8070c80 0x7f3fe80730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:36.419 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.416+0000 7f3ff37fe640 1 -- 172.21.15.82:0/776990707 --> [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3fe8073750 con 0x7f3fe8070c80 2024-06-06T07:41:36.419 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.416+0000 7f3ff37fe640 1 -- 172.21.15.82:0/776990707 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fec0ab4d0 con 0x7f3ffc0ff3b0 2024-06-06T07:41:36.419 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.416+0000 7f3ffad9d640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] conn(0x7f3fe8070c80 0x7f3fe80730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:36.419 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.417+0000 7f3ff9d9b640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f3fe806aef0 0x7f3fe806d3b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc195650 tx=0x7f3fe400b390 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:36.419 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.417+0000 7f3ffad9d640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] conn(0x7f3fe8070c80 0x7f3fe80730a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:36.419 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.417+0000 7f3ff37fe640 1 -- 172.21.15.82:0/776990707 <== osd.0 v2:172.21.15.82:6802/879999521 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f3fe8073750 con 0x7f3fe8070c80 2024-06-06T07:41:36.431 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.431+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 --> [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3fdc002c20 con 0x7f3fe8070c80 2024-06-06T07:41:36.431 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.431+0000 7f3ff37fe640 1 -- 172.21.15.82:0/776990707 <== osd.0 v2:172.21.15.82:6802/879999521 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3fdc002c20 con 0x7f3fe8070c80 2024-06-06T07:41:36.432 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.431+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] conn(0x7f3fe8070c80 msgr2=0x7f3fe80730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:36.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.431+0000 7f40008f8640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] conn(0x7f3fe8070c80 0x7f3fe80730a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f3fe806aef0 msgr2=0x7f3fe806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:36.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f3fe806aef0 0x7f3fe806d3b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc195650 tx=0x7f3fe400b390 comp rx=0 tx=0).stop 2024-06-06T07:41:36.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 msgr2=0x7f3ffc193f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:36.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 0x7f3ffc193f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3fec02a9c0 tx=0x7f3fec0055a0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 shutdown_connections 2024-06-06T07:41:36.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6802/879999521,v1:172.21.15.82:6803/879999521] conn(0x7f3fe8070c80 0x7f3fe80730a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f3fe806aef0 0x7f3fe806d3b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ffc0ffd60 0x7f3ffc194470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 --2- 172.21.15.82:0/776990707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ffc0ff3b0 0x7f3ffc193f30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 >> 172.21.15.82:0/776990707 conn(0x7f3ffc0fac80 msgr2=0x7f3ffc0fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:36.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 shutdown_connections 2024-06-06T07:41:36.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.432+0000 7f40008f8640 1 -- 172.21.15.82:0/776990707 wait complete. 2024-06-06T07:41:36.990 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:36 smithi082 ceph-mon[29354]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:36.993 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.990+0000 7fc84c86e640 1 -- 172.21.15.82:0/3572673198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100460 msgr2=0x7fc848100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:36.993 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.990+0000 7fc84c86e640 1 --2- 172.21.15.82:0/3572673198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100460 0x7fc848100840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc840009c20 tx=0x7fc84002a970 comp rx=0 tx=0).stop 2024-06-06T07:41:36.993 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.990+0000 7fc84c86e640 1 -- 172.21.15.82:0/3572673198 shutdown_connections 2024-06-06T07:41:36.993 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.990+0000 7fc84c86e640 1 --2- 172.21.15.82:0/3572673198 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc848100e10 0x7fc848104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.990+0000 7fc84c86e640 1 --2- 172.21.15.82:0/3572673198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100460 0x7fc848100840 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.990+0000 7fc84c86e640 1 -- 172.21.15.82:0/3572673198 >> 172.21.15.82:0/3572673198 conn(0x7fc8480fbd10 msgr2=0x7fc8480fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.990+0000 7fc84c86e640 1 -- 172.21.15.82:0/3572673198 shutdown_connections 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.990+0000 7fc84c86e640 1 -- 172.21.15.82:0/3572673198 wait complete. 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc84c86e640 1 Processor -- start 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc84c86e640 1 -- start start 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc84c86e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc848100460 0x7fc8480727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc84c86e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100e10 0x7fc848072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc84c86e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc848073230 con 0x7fc848100e10 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc84c86e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc848073370 con 0x7fc848100460 2024-06-06T07:41:36.994 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc845d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100e10 0x7fc848072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc845d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100e10 0x7fc848072cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:58270/0 (socket says 172.21.15.82:58270) 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc845d9b640 1 -- 172.21.15.82:0/1252224526 learned_addr learned my addr 172.21.15.82:0/1252224526 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc84659c640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc848100460 0x7fc8480727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc845d9b640 1 -- 172.21.15.82:0/1252224526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc848100460 msgr2=0x7fc8480727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc845d9b640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc848100460 0x7fc8480727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.991+0000 7fc845d9b640 1 -- 172.21.15.82:0/1252224526 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc840009900 con 0x7fc848100e10 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.992+0000 7fc845d9b640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100e10 0x7fc848072cf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc83000d6c0 tx=0x7fc83000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.992+0000 7fc83f7fe640 1 -- 172.21.15.82:0/1252224526 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8300051f0 con 0x7fc848100e10 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.992+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8481a2410 con 0x7fc848100e10 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.992+0000 7fc83f7fe640 1 -- 172.21.15.82:0/1252224526 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc830009ce0 con 0x7fc848100e10 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.992+0000 7fc83f7fe640 1 -- 172.21.15.82:0/1252224526 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc830010460 con 0x7fc848100e10 2024-06-06T07:41:36.995 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.992+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8481a2770 con 0x7fc848100e10 2024-06-06T07:41:36.996 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.993+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc84810d730 con 0x7fc848100e10 2024-06-06T07:41:36.997 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.993+0000 7fc83f7fe640 1 -- 172.21.15.82:0/1252224526 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fc830005350 con 0x7fc848100e10 2024-06-06T07:41:36.997 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.994+0000 7fc83f7fe640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc83406aef0 0x7fc83406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:36.997 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.994+0000 7fc83f7fe640 1 -- 172.21.15.82:0/1252224526 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc830089630 con 0x7fc848100e10 2024-06-06T07:41:36.997 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.994+0000 7fc84659c640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc83406aef0 0x7fc83406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:36.997 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.994+0000 7fc83f7fe640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] conn(0x7fc834070c80 0x7fc8340730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:36.997 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.994+0000 7fc83f7fe640 1 -- 172.21.15.82:0/1252224526 --> [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc834073750 con 0x7fc834070c80 2024-06-06T07:41:36.997 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.994+0000 7fc83f7fe640 1 -- 172.21.15.82:0/1252224526 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc83008bd00 con 0x7fc848100e10 2024-06-06T07:41:36.997 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.994+0000 7fc846d9d640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] conn(0x7fc834070c80 0x7fc8340730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:36.997 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.995+0000 7fc84659c640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc83406aef0 0x7fc83406d3b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc840000f80 tx=0x7fc84003d040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:36.998 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.995+0000 7fc846d9d640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] conn(0x7fc834070c80 0x7fc8340730a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:36.998 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:36.995+0000 7fc83f7fe640 1 -- 172.21.15.82:0/1252224526 <== osd.1 v2:172.21.15.82:6810/1482740538 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fc834073750 con 0x7fc834070c80 2024-06-06T07:41:37.009 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.009+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 --> [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc848060b30 con 0x7fc834070c80 2024-06-06T07:41:37.009 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.009+0000 7fc83f7fe640 1 -- 172.21.15.82:0/1252224526 <== osd.1 v2:172.21.15.82:6810/1482740538 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc848060b30 con 0x7fc834070c80 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] conn(0x7fc834070c80 msgr2=0x7fc8340730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] conn(0x7fc834070c80 0x7fc8340730a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc83406aef0 msgr2=0x7fc83406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc83406aef0 0x7fc83406d3b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc840000f80 tx=0x7fc84003d040 comp rx=0 tx=0).stop 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100e10 msgr2=0x7fc848072cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100e10 0x7fc848072cf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc83000d6c0 tx=0x7fc83000db80 comp rx=0 tx=0).stop 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 shutdown_connections 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6810/1482740538,v1:172.21.15.82:6811/1482740538] conn(0x7fc834070c80 0x7fc8340730a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc83406aef0 0x7fc83406d3b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc848100e10 0x7fc848072cf0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 --2- 172.21.15.82:0/1252224526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc848100460 0x7fc8480727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 >> 172.21.15.82:0/1252224526 conn(0x7fc8480fbd10 msgr2=0x7fc8480fc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 shutdown_connections 2024-06-06T07:41:37.012 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.010+0000 7fc84c86e640 1 -- 172.21.15.82:0/1252224526 wait complete. 2024-06-06T07:41:37.092 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:36 smithi184 ceph-mon[37288]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:37.282 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.256+0000 7f6f5e13b640 1 -- 172.21.15.82:0/2363170305 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ff360 msgr2=0x7f6f580ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.282 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.256+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/2363170305 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ff360 0x7f6f580ff740 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009880 tx=0x7f6f4802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.282 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.257+0000 7f6f5e13b640 1 -- 172.21.15.82:0/2363170305 shutdown_connections 2024-06-06T07:41:37.282 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.257+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/2363170305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f580ffd10 0x7f6f58103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.282 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.257+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/2363170305 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ff360 0x7f6f580ff740 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.257+0000 7f6f5e13b640 1 -- 172.21.15.82:0/2363170305 >> 172.21.15.82:0/2363170305 conn(0x7f6f58075e10 msgr2=0x7f6f580fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.257+0000 7f6f5e13b640 1 -- 172.21.15.82:0/2363170305 shutdown_connections 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.257+0000 7f6f5e13b640 1 -- 172.21.15.82:0/2363170305 wait complete. 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.257+0000 7f6f5e13b640 1 Processor -- start 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.257+0000 7f6f5e13b640 1 -- start start 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f5e13b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f580ff360 0x7f6f58198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f5e13b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ffd10 0x7f6f581989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f5e13b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58199030 con 0x7f6f580ffd10 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f5e13b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5819cd50 con 0x7f6f580ff360 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f56ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ffd10 0x7f6f581989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f56ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ffd10 0x7f6f581989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:58294/0 (socket says 172.21.15.82:58294) 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f56ffd640 1 -- 172.21.15.82:0/285154549 learned_addr learned my addr 172.21.15.82:0/285154549 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:37.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f577fe640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f580ff360 0x7f6f58198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f56ffd640 1 -- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f580ff360 msgr2=0x7f6f58198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f56ffd640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f580ff360 0x7f6f58198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f56ffd640 1 -- 172.21.15.82:0/285154549 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48009560 con 0x7f6f580ffd10 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f56ffd640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ffd10 0x7f6f581989a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00d6c0 tx=0x7f6f4c00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f54ff9640 1 -- 172.21.15.82:0/285154549 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f4c0051f0 con 0x7f6f580ffd10 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5819d030 con 0x7f6f580ffd10 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f54ff9640 1 -- 172.21.15.82:0/285154549 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f4c009ce0 con 0x7f6f580ffd10 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f54ff9640 1 -- 172.21.15.82:0/285154549 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f4c010460 con 0x7f6f580ffd10 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.258+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5819d580 con 0x7f6f580ffd10 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.259+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6f5810c6b0 con 0x7f6f580ffd10 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.260+0000 7f6f54ff9640 1 -- 172.21.15.82:0/285154549 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f6f4c005350 con 0x7f6f580ffd10 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.260+0000 7f6f54ff9640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6f4006aef0 0x7f6f4006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:37.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.260+0000 7f6f54ff9640 1 -- 172.21.15.82:0/285154549 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6f4c089a10 con 0x7f6f580ffd10 2024-06-06T07:41:37.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.260+0000 7f6f577fe640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6f4006aef0 0x7f6f4006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:37.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.261+0000 7f6f54ff9640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] conn(0x7f6f40070c80 0x7f6f400730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:37.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.261+0000 7f6f54ff9640 1 -- 172.21.15.82:0/285154549 --> [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f40073750 con 0x7f6f40070c80 2024-06-06T07:41:37.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.261+0000 7f6f54ff9640 1 -- 172.21.15.82:0/285154549 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f4c089e70 con 0x7f6f580ffd10 2024-06-06T07:41:37.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.261+0000 7f6f57fff640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] conn(0x7f6f40070c80 0x7f6f400730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:37.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.261+0000 7f6f577fe640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6f4006aef0 0x7f6f4006d3b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009230 tx=0x7f6f480058d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:37.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.261+0000 7f6f57fff640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] conn(0x7f6f40070c80 0x7f6f400730a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.262+0000 7f6f54ff9640 1 -- 172.21.15.82:0/285154549 <== osd.3 v2:172.21.15.184:6800/3221286727 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f6f40073750 con 0x7f6f40070c80 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.276+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 --> [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6f5819daa0 con 0x7f6f40070c80 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f54ff9640 1 -- 172.21.15.82:0/285154549 <== osd.3 v2:172.21.15.184:6800/3221286727 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6f5819daa0 con 0x7f6f40070c80 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] conn(0x7f6f40070c80 msgr2=0x7f6f400730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] conn(0x7f6f40070c80 0x7f6f400730a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6f4006aef0 msgr2=0x7f6f4006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6f4006aef0 0x7f6f4006d3b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009230 tx=0x7f6f480058d0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ffd10 msgr2=0x7f6f581989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ffd10 0x7f6f581989a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00d6c0 tx=0x7f6f4c00db80 comp rx=0 tx=0).stop 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 shutdown_connections 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:6800/3221286727,v1:172.21.15.184:6801/3221286727] conn(0x7f6f40070c80 0x7f6f400730a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6f4006aef0 0x7f6f4006d3b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.287 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f580ffd10 0x7f6f581989a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.287 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 --2- 172.21.15.82:0/285154549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6f580ff360 0x7f6f58198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.287 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 >> 172.21.15.82:0/285154549 conn(0x7f6f58075e10 msgr2=0x7f6f58103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:37.287 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 shutdown_connections 2024-06-06T07:41:37.287 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.277+0000 7f6f5e13b640 1 -- 172.21.15.82:0/285154549 wait complete. 2024-06-06T07:41:37.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 -- 172.21.15.82:0/4045182105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 msgr2=0x7f739c103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 --2- 172.21.15.82:0/4045182105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 0x7f739c103da0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7390009880 tx=0x7f739002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 -- 172.21.15.82:0/4045182105 shutdown_connections 2024-06-06T07:41:37.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 --2- 172.21.15.82:0/4045182105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 0x7f739c103da0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 --2- 172.21.15.82:0/4045182105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f739c0ff3a0 0x7f739c0ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 -- 172.21.15.82:0/4045182105 >> 172.21.15.82:0/4045182105 conn(0x7f739c0fac70 msgr2=0x7f739c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:37.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 -- 172.21.15.82:0/4045182105 shutdown_connections 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 -- 172.21.15.82:0/4045182105 wait complete. 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 Processor -- start 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.467+0000 7f73a26d4640 1 -- start start 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f73a26d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f739c0ff3a0 0x7f739c1962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f73a26d4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 0x7f739c1967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f73a26d4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c196e80 con 0x7f739c0ffd50 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f73a26d4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c19aba0 con 0x7f739c0ff3a0 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f739b7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 0x7f739c1967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f739b7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 0x7f739c1967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:58314/0 (socket says 172.21.15.82:58314) 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f739b7fe640 1 -- 172.21.15.82:0/1543705445 learned_addr learned my addr 172.21.15.82:0/1543705445 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f739b7fe640 1 -- 172.21.15.82:0/1543705445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f739c0ff3a0 msgr2=0x7f739c1962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f739b7fe640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f739c0ff3a0 0x7f739c1962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f739b7fe640 1 -- 172.21.15.82:0/1543705445 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7390009560 con 0x7f739c0ffd50 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f739b7fe640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 0x7f739c1967f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f739002a9c0 tx=0x7f73900055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.468+0000 7f73997fa640 1 -- 172.21.15.82:0/1543705445 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7390004aa0 con 0x7f739c0ffd50 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.469+0000 7f73997fa640 1 -- 172.21.15.82:0/1543705445 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7390031470 con 0x7f739c0ffd50 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.469+0000 7f73997fa640 1 -- 172.21.15.82:0/1543705445 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73900039e0 con 0x7f739c0ffd50 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.469+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c19ae20 con 0x7f739c0ffd50 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.469+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c19b310 con 0x7f739c0ffd50 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.469+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f739c10c650 con 0x7f739c0ffd50 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.470+0000 7f73997fa640 1 -- 172.21.15.82:0/1543705445 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7390031c00 con 0x7f739c0ffd50 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.470+0000 7f73997fa640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f738406aef0 0x7f738406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.471+0000 7f73997fa640 1 -- 172.21.15.82:0/1543705445 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f73900aacc0 con 0x7f739c0ffd50 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.471+0000 7f739bfff640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f738406aef0 0x7f738406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:37.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.471+0000 7f73997fa640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] conn(0x7f7384070c80 0x7f73840730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:37.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.471+0000 7f73997fa640 1 -- 172.21.15.82:0/1543705445 --> [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7384073750 con 0x7f7384070c80 2024-06-06T07:41:37.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.471+0000 7f73997fa640 1 -- 172.21.15.82:0/1543705445 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73900ae020 con 0x7f739c0ffd50 2024-06-06T07:41:37.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.471+0000 7f73a0c71640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] conn(0x7f7384070c80 0x7f73840730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:37.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.471+0000 7f739bfff640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f738406aef0 0x7f738406d3b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f738c005fd0 tx=0x7f738c005850 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:37.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.471+0000 7f73a0c71640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] conn(0x7f7384070c80 0x7f73840730a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:37.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.472+0000 7f73997fa640 1 -- 172.21.15.82:0/1543705445 <== osd.5 v2:172.21.15.184:6816/2823055081 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f7384073750 con 0x7f7384070c80 2024-06-06T07:41:37.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.486+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 --> [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f739c19b770 con 0x7f7384070c80 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.486+0000 7f73997fa640 1 -- 172.21.15.82:0/1543705445 <== osd.5 v2:172.21.15.184:6816/2823055081 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f739c19b770 con 0x7f7384070c80 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 >> [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] conn(0x7f7384070c80 msgr2=0x7f73840730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] conn(0x7f7384070c80 0x7f73840730a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f738406aef0 msgr2=0x7f738406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f738406aef0 0x7f738406d3b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f738c005fd0 tx=0x7f738c005850 comp rx=0 tx=0).stop 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 msgr2=0x7f739c1967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 0x7f739c1967f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f739002a9c0 tx=0x7f73900055a0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 shutdown_connections 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.184:6816/2823055081,v1:172.21.15.184:6817/2823055081] conn(0x7f7384070c80 0x7f73840730a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f738406aef0 0x7f738406d3b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f739c0ffd50 0x7f739c1967f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 --2- 172.21.15.82:0/1543705445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f739c0ff3a0 0x7f739c1962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:37.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 >> 172.21.15.82:0/1543705445 conn(0x7f739c0fac70 msgr2=0x7f739c068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:37.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 shutdown_connections 2024-06-06T07:41:37.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:37.487+0000 7f73a26d4640 1 -- 172.21.15.82:0/1543705445 wait complete. 2024-06-06T07:41:37.983 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:41:38.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:41:38.386 INFO:teuthology.orchestra.run.smithi082.stdout:167503724554 2024-06-06T07:41:38.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-06T07:41:38.939 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:38 smithi082 ceph-mon[29354]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:39.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:38 smithi184 ceph-mon[37288]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:39.247 INFO:teuthology.orchestra.run.smithi082.stdout:60129542174 2024-06-06T07:41:39.248 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-06T07:41:39.695 INFO:teuthology.orchestra.run.smithi082.stdout:111669149715 2024-06-06T07:41:39.696 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-06T07:41:39.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.901+0000 7fd0b5f56640 1 -- 172.21.15.82:0/3627779487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101d30 msgr2=0x7fd0b010a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:39.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.901+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/3627779487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101d30 0x7fd0b010a220 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009880 tx=0x7fd0a402a530 comp rx=0 tx=0).stop 2024-06-06T07:41:39.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.901+0000 7fd0b5f56640 1 -- 172.21.15.82:0/3627779487 shutdown_connections 2024-06-06T07:41:39.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.901+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/3627779487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101d30 0x7fd0b010a220 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:39.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.901+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/3627779487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0b0101410 0x7fd0b01017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:39.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.901+0000 7fd0b5f56640 1 -- 172.21.15.82:0/3627779487 >> 172.21.15.82:0/3627779487 conn(0x7fd0b00fac80 msgr2=0x7fd0b00fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:39.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.901+0000 7fd0b5f56640 1 -- 172.21.15.82:0/3627779487 shutdown_connections 2024-06-06T07:41:39.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.901+0000 7fd0b5f56640 1 -- 172.21.15.82:0/3627779487 wait complete. 2024-06-06T07:41:39.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.902+0000 7fd0b5f56640 1 Processor -- start 2024-06-06T07:41:39.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.902+0000 7fd0b5f56640 1 -- start start 2024-06-06T07:41:39.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.902+0000 7fd0b5f56640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101410 0x7fd0b0196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:39.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.902+0000 7fd0b5f56640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0b0101d30 0x7fd0b0196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:39.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.902+0000 7fd0b5f56640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b0196f00 con 0x7fd0b0101410 2024-06-06T07:41:39.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.902+0000 7fd0b5f56640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b019ac20 con 0x7fd0b0101d30 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0af7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101410 0x7fd0b0196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0af7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101410 0x7fd0b0196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:58332/0 (socket says 172.21.15.82:58332) 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0af7fe640 1 -- 172.21.15.82:0/1703142490 learned_addr learned my addr 172.21.15.82:0/1703142490 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0aeffd640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0b0101d30 0x7fd0b0196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0af7fe640 1 -- 172.21.15.82:0/1703142490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0b0101d30 msgr2=0x7fd0b0196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0af7fe640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0b0101d30 0x7fd0b0196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0af7fe640 1 -- 172.21.15.82:0/1703142490 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a4009560 con 0x7fd0b0101410 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0af7fe640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101410 0x7fd0b0196330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000b420 tx=0x7fd0a000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0acff9640 1 -- 172.21.15.82:0/1703142490 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0a0011800 con 0x7fd0b0101410 2024-06-06T07:41:39.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b019af00 con 0x7fd0b0101410 2024-06-06T07:41:39.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0acff9640 1 -- 172.21.15.82:0/1703142490 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0a0011e20 con 0x7fd0b0101410 2024-06-06T07:41:39.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0acff9640 1 -- 172.21.15.82:0/1703142490 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0a000f2e0 con 0x7fd0b0101410 2024-06-06T07:41:39.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.903+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b019b450 con 0x7fd0b0101410 2024-06-06T07:41:39.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.904+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd0b010d860 con 0x7fd0b0101410 2024-06-06T07:41:39.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.905+0000 7fd0acff9640 1 -- 172.21.15.82:0/1703142490 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fd0a0010380 con 0x7fd0b0101410 2024-06-06T07:41:39.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.905+0000 7fd0acff9640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd09806f300 0x7fd0980717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:39.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.905+0000 7fd0acff9640 1 -- 172.21.15.82:0/1703142490 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd0a0089570 con 0x7fd0b0101410 2024-06-06T07:41:39.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.905+0000 7fd0aeffd640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd09806f300 0x7fd0980717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:39.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.905+0000 7fd0acff9640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] conn(0x7fd098075090 0x7fd0980774b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:39.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.905+0000 7fd0acff9640 1 -- 172.21.15.82:0/1703142490 --> [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd098077b60 con 0x7fd098075090 2024-06-06T07:41:39.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.905+0000 7fd0acff9640 1 -- 172.21.15.82:0/1703142490 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0a008bc40 con 0x7fd0b0101410 2024-06-06T07:41:39.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.906+0000 7fd0affff640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] conn(0x7fd098075090 0x7fd0980774b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:39.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.906+0000 7fd0aeffd640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd09806f300 0x7fd0980717c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4000c00 tx=0x7fd0a4004120 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:39.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.906+0000 7fd0affff640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] conn(0x7fd098075090 0x7fd0980774b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:39.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.907+0000 7fd0acff9640 1 -- 172.21.15.82:0/1703142490 <== osd.2 v2:172.21.15.82:6818/1581462724 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fd098077b60 con 0x7fd098075090 2024-06-06T07:41:39.923 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.923+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 --> [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd0b0069060 con 0x7fd098075090 2024-06-06T07:41:39.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.923+0000 7fd0acff9640 1 -- 172.21.15.82:0/1703142490 <== osd.2 v2:172.21.15.82:6818/1581462724 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd0b0069060 con 0x7fd098075090 2024-06-06T07:41:39.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] conn(0x7fd098075090 msgr2=0x7fd0980774b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:39.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] conn(0x7fd098075090 0x7fd0980774b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:39.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd09806f300 msgr2=0x7fd0980717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:39.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd09806f300 0x7fd0980717c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4000c00 tx=0x7fd0a4004120 comp rx=0 tx=0).stop 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101410 msgr2=0x7fd0b0196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101410 0x7fd0b0196330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000b420 tx=0x7fd0a000b8e0 comp rx=0 tx=0).stop 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 shutdown_connections 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6818/1581462724,v1:172.21.15.82:6819/1581462724] conn(0x7fd098075090 0x7fd0980774b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd09806f300 0x7fd0980717c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0b0101d30 0x7fd0b0196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 --2- 172.21.15.82:0/1703142490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0b0101410 0x7fd0b0196330 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 >> 172.21.15.82:0/1703142490 conn(0x7fd0b00fac80 msgr2=0x7fd0b0102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 shutdown_connections 2024-06-06T07:41:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:39.924+0000 7fd0b5f56640 1 -- 172.21.15.82:0/1703142490 wait complete. 2024-06-06T07:41:40.693 INFO:teuthology.orchestra.run.smithi082.stdout:81604378649 2024-06-06T07:41:40.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-06T07:41:40.843 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:40 smithi184 ceph-mon[37288]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:40.998 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:40 smithi082 ceph-mon[29354]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:41.108 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:41.108 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:41.136 INFO:teuthology.orchestra.run.smithi082.stdout:38654705699 2024-06-06T07:41:41.136 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-06T07:41:41.153 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:41.788 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:41.874 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:41 smithi082 ceph-mon[29354]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:42.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:41 smithi184 ceph-mon[37288]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:44.191 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:44.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:44 smithi082 ceph-mon[29354]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:45.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:44 smithi184 ceph-mon[37288]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:45.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.795+0000 7f5469425640 1 -- 172.21.15.82:0/131548784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 msgr2=0x7f5464104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:45.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.795+0000 7f5469425640 1 --2- 172.21.15.82:0/131548784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 0x7f5464104e70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5458009c20 tx=0x7f545802a970 comp rx=0 tx=0).stop 2024-06-06T07:41:45.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.795+0000 7f5469425640 1 -- 172.21.15.82:0/131548784 shutdown_connections 2024-06-06T07:41:45.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.795+0000 7f5469425640 1 --2- 172.21.15.82:0/131548784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 0x7f5464104e70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.795+0000 7f5469425640 1 --2- 172.21.15.82:0/131548784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5464100470 0x7f5464100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.795+0000 7f5469425640 1 -- 172.21.15.82:0/131548784 >> 172.21.15.82:0/131548784 conn(0x7f54640fbd20 msgr2=0x7f54640fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.795+0000 7f5469425640 1 -- 172.21.15.82:0/131548784 shutdown_connections 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.795+0000 7f5469425640 1 -- 172.21.15.82:0/131548784 wait complete. 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.795+0000 7f5469425640 1 Processor -- start 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f5469425640 1 -- start start 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f5469425640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5464100470 0x7f5464196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f5469425640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 0x7f5464196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f5469425640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464196ee0 con 0x7f5464100e20 2024-06-06T07:41:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f5469425640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546419ac00 con 0x7f5464100470 2024-06-06T07:41:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f54627fc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 0x7f5464196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f54627fc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 0x7f5464196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:60294/0 (socket says 172.21.15.82:60294) 2024-06-06T07:41:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f54627fc640 1 -- 172.21.15.82:0/2233397489 learned_addr learned my addr 172.21.15.82:0/2233397489 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f5462ffd640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5464100470 0x7f5464196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f54627fc640 1 -- 172.21.15.82:0/2233397489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5464100470 msgr2=0x7f5464196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f54627fc640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5464100470 0x7f5464196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f54627fc640 1 -- 172.21.15.82:0/2233397489 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5458009900 con 0x7f5464100e20 2024-06-06T07:41:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f54627fc640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 0x7f5464196850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f545802ae90 tx=0x7f54580308a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.796+0000 7f544bfff640 1 -- 172.21.15.82:0/2233397489 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f545802f590 con 0x7f5464100e20 2024-06-06T07:41:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.797+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546419ae80 con 0x7f5464100e20 2024-06-06T07:41:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.797+0000 7f544bfff640 1 -- 172.21.15.82:0/2233397489 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5458030ca0 con 0x7f5464100e20 2024-06-06T07:41:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.797+0000 7f544bfff640 1 -- 172.21.15.82:0/2233397489 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5458039680 con 0x7f5464100e20 2024-06-06T07:41:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.797+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546419b370 con 0x7f5464100e20 2024-06-06T07:41:45.801 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.798+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5464101950 con 0x7f5464100e20 2024-06-06T07:41:45.801 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.798+0000 7f544bfff640 1 -- 172.21.15.82:0/2233397489 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f545800a040 con 0x7f5464100e20 2024-06-06T07:41:45.801 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.798+0000 7f544bfff640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f544c06aef0 0x7f544c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:45.801 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.798+0000 7f544bfff640 1 -- 172.21.15.82:0/2233397489 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f545802c040 con 0x7f5464100e20 2024-06-06T07:41:45.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.799+0000 7f5462ffd640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f544c06aef0 0x7f544c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:45.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.799+0000 7f5462ffd640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f544c06aef0 0x7f544c06d3b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f545c00a810 tx=0x7f545c008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:45.803 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.803+0000 7f544bfff640 1 -- 172.21.15.82:0/2233397489 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f54580766d0 con 0x7f5464100e20 2024-06-06T07:41:45.979 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.978+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f546410d500 con 0x7f5464100e20 2024-06-06T07:41:45.979 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.978+0000 7f544bfff640 1 -- 172.21.15.82:0/2233397489 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5458079cf0 con 0x7f5464100e20 2024-06-06T07:41:45.979 INFO:teuthology.orchestra.run.smithi082.stdout:167503724556 2024-06-06T07:41:45.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f544c06aef0 msgr2=0x7f544c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:45.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f544c06aef0 0x7f544c06d3b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f545c00a810 tx=0x7f545c008040 comp rx=0 tx=0).stop 2024-06-06T07:41:45.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 msgr2=0x7f5464196850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:45.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 0x7f5464196850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f545802ae90 tx=0x7f54580308a0 comp rx=0 tx=0).stop 2024-06-06T07:41:45.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 shutdown_connections 2024-06-06T07:41:45.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f544c06aef0 0x7f544c06d3b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:45.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5464100e20 0x7f5464196850 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:45.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 --2- 172.21.15.82:0/2233397489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5464100470 0x7f5464196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:45.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 >> 172.21.15.82:0/2233397489 conn(0x7f54640fbd20 msgr2=0x7f54640fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:45.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 shutdown_connections 2024-06-06T07:41:45.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:45.981+0000 7f5469425640 1 -- 172.21.15.82:0/2233397489 wait complete. 2024-06-06T07:41:46.007 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.006+0000 7f6620bc6640 1 -- 172.21.15.82:0/1408934865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0fef70 msgr2=0x7f661c0ff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:46.007 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.006+0000 7f6620bc6640 1 --2- 172.21.15.82:0/1408934865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0fef70 0x7f661c0ff350 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f660c0098b0 tx=0x7f660c02a4d0 comp rx=0 tx=0).stop 2024-06-06T07:41:46.007 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.007+0000 7f6620bc6640 1 -- 172.21.15.82:0/1408934865 shutdown_connections 2024-06-06T07:41:46.007 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.007+0000 7f6620bc6640 1 --2- 172.21.15.82:0/1408934865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0ff890 0x7f661c10c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:46.007 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.007+0000 7f6620bc6640 1 --2- 172.21.15.82:0/1408934865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0fef70 0x7f661c0ff350 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:46.008 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.007+0000 7f6620bc6640 1 -- 172.21.15.82:0/1408934865 >> 172.21.15.82:0/1408934865 conn(0x7f661c0fac30 msgr2=0x7f661c0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:46.008 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.007+0000 7f6620bc6640 1 -- 172.21.15.82:0/1408934865 shutdown_connections 2024-06-06T07:41:46.008 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.007+0000 7f6620bc6640 1 -- 172.21.15.82:0/1408934865 wait complete. 2024-06-06T07:41:46.010 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f6620bc6640 1 Processor -- start 2024-06-06T07:41:46.010 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f6620bc6640 1 -- start start 2024-06-06T07:41:46.010 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f6620bc6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0fef70 0x7f661c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:46.010 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f6620bc6640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0ff890 0x7f661c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:46.010 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f6620bc6640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c199110 con 0x7f661c0ff890 2024-06-06T07:41:46.010 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f6620bc6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c19ce30 con 0x7f661c0fef70 2024-06-06T07:41:46.010 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f6619d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0ff890 0x7f661c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f6619d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0ff890 0x7f661c198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:60324/0 (socket says 172.21.15.82:60324) 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f6619d9b640 1 -- 172.21.15.82:0/662460270 learned_addr learned my addr 172.21.15.82:0/662460270 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.008+0000 7f661a59c640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0fef70 0x7f661c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f6619d9b640 1 -- 172.21.15.82:0/662460270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0fef70 msgr2=0x7f661c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f6619d9b640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0fef70 0x7f661c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f6619d9b640 1 -- 172.21.15.82:0/662460270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c009560 con 0x7f661c0ff890 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f661a59c640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0fef70 0x7f661c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f6619d9b640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0ff890 0x7f661c198a80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f661000b420 tx=0x7f661000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f660b7fe640 1 -- 172.21.15.82:0/662460270 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6610011800 con 0x7f661c0ff890 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f660b7fe640 1 -- 172.21.15.82:0/662460270 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6610011e20 con 0x7f661c0ff890 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f660b7fe640 1 -- 172.21.15.82:0/662460270 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f661000f2e0 con 0x7f661c0ff890 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c19d110 con 0x7f661c0ff890 2024-06-06T07:41:46.011 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.009+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c19d600 con 0x7f661c0ff890 2024-06-06T07:41:46.012 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.011+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c068da0 con 0x7f661c0ff890 2024-06-06T07:41:46.012 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.011+0000 7f660b7fe640 1 -- 172.21.15.82:0/662460270 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f6610010380 con 0x7f661c0ff890 2024-06-06T07:41:46.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.012+0000 7f660b7fe640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f660406aef0 0x7f660406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:46.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.012+0000 7f660b7fe640 1 -- 172.21.15.82:0/662460270 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6610089410 con 0x7f661c0ff890 2024-06-06T07:41:46.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.012+0000 7f661a59c640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f660406aef0 0x7f660406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:46.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.012+0000 7f661a59c640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f660406aef0 0x7f660406d3b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f660c009880 tx=0x7f660c0058d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:46.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.016+0000 7f660b7fe640 1 -- 172.21.15.82:0/662460270 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6610054a70 con 0x7f661c0ff890 2024-06-06T07:41:46.193 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.193+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f661c04e210 con 0x7f661c0ff890 2024-06-06T07:41:46.194 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.193+0000 7f660b7fe640 1 -- 172.21.15.82:0/662460270 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6610058090 con 0x7f661c0ff890 2024-06-06T07:41:46.194 INFO:teuthology.orchestra.run.smithi082.stdout:146028888080 2024-06-06T07:41:46.195 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f660406aef0 msgr2=0x7f660406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:46.195 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f660406aef0 0x7f660406d3b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f660c009880 tx=0x7f660c0058d0 comp rx=0 tx=0).stop 2024-06-06T07:41:46.195 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0ff890 msgr2=0x7f661c198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:46.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0ff890 0x7f661c198a80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f661000b420 tx=0x7f661000b8e0 comp rx=0 tx=0).stop 2024-06-06T07:41:46.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 shutdown_connections 2024-06-06T07:41:46.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f660406aef0 0x7f660406d3b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:46.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f661c0ff890 0x7f661c198a80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:46.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 --2- 172.21.15.82:0/662460270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f661c0fef70 0x7f661c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:46.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 >> 172.21.15.82:0/662460270 conn(0x7f661c0fac30 msgr2=0x7f661c0fc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:46.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 shutdown_connections 2024-06-06T07:41:46.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:46.195+0000 7f6620bc6640 1 -- 172.21.15.82:0/662460270 wait complete. 2024-06-06T07:41:46.912 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:46 smithi082 ceph-mon[29354]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:46.912 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:46 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2233397489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-06T07:41:46.912 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:46 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/662460270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-06T07:41:47.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:46 smithi184 ceph-mon[37288]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:47.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:46 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2233397489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-06T07:41:47.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:46 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/662460270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-06T07:41:47.447 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-06-06T07:41:47.447 DEBUG:teuthology.parallel:result is None 2024-06-06T07:41:48.547 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.538+0000 7f98d3687640 1 -- 172.21.15.82:0/978946889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 msgr2=0x7f98cc10a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.538+0000 7f98d3687640 1 --2- 172.21.15.82:0/978946889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 0x7f98cc10a270 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f98c00098b0 tx=0x7f98c002a560 comp rx=0 tx=0).stop 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.538+0000 7f98d3687640 1 -- 172.21.15.82:0/978946889 shutdown_connections 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.538+0000 7f98d3687640 1 --2- 172.21.15.82:0/978946889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 0x7f98cc10a270 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.538+0000 7f98d3687640 1 --2- 172.21.15.82:0/978946889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98cc101460 0x7f98cc101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.538+0000 7f98d3687640 1 -- 172.21.15.82:0/978946889 >> 172.21.15.82:0/978946889 conn(0x7f98cc075e10 msgr2=0x7f98cc0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.538+0000 7f98d3687640 1 -- 172.21.15.82:0/978946889 shutdown_connections 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.538+0000 7f98d3687640 1 -- 172.21.15.82:0/978946889 wait complete. 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.539+0000 7f98d3687640 1 Processor -- start 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.539+0000 7f98d3687640 1 -- start start 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.539+0000 7f98d3687640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98cc101460 0x7f98cc1962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:48.548 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.539+0000 7f98d3687640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 0x7f98cc196810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.539+0000 7f98d3687640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc196ea0 con 0x7f98cc101d80 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.539+0000 7f98d3687640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc19abc0 con 0x7f98cc101460 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.539+0000 7f98d0c22640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 0x7f98cc196810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98d0c22640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 0x7f98cc196810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:60342/0 (socket says 172.21.15.82:60342) 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98d0c22640 1 -- 172.21.15.82:0/1203091195 learned_addr learned my addr 172.21.15.82:0/1203091195 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98d1423640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98cc101460 0x7f98cc1962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98d0c22640 1 -- 172.21.15.82:0/1203091195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98cc101460 msgr2=0x7f98cc1962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98d0c22640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98cc101460 0x7f98cc1962d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98d0c22640 1 -- 172.21.15.82:0/1203091195 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0009560 con 0x7f98cc101d80 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98d0c22640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 0x7f98cc196810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f98c0006010 tx=0x7f98c00055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98ca7fc640 1 -- 172.21.15.82:0/1203091195 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c0004aa0 con 0x7f98cc101d80 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc19ae40 con 0x7f98cc101d80 2024-06-06T07:41:48.549 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98ca7fc640 1 -- 172.21.15.82:0/1203091195 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98c0031470 con 0x7f98cc101d80 2024-06-06T07:41:48.550 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98ca7fc640 1 -- 172.21.15.82:0/1203091195 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c0003940 con 0x7f98cc101d80 2024-06-06T07:41:48.550 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.540+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc19b330 con 0x7f98cc101d80 2024-06-06T07:41:48.550 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.541+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc068470 con 0x7f98cc101d80 2024-06-06T07:41:48.550 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.542+0000 7f98ca7fc640 1 -- 172.21.15.82:0/1203091195 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f98c0031b90 con 0x7f98cc101d80 2024-06-06T07:41:48.550 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.542+0000 7f98ca7fc640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f98b406aef0 0x7f98b406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:48.550 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.542+0000 7f98ca7fc640 1 -- 172.21.15.82:0/1203091195 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f98c00aac80 con 0x7f98cc101d80 2024-06-06T07:41:48.550 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.542+0000 7f98d1423640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f98b406aef0 0x7f98b406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:48.550 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.542+0000 7f98d1423640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f98b406aef0 0x7f98b406d3b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f98bc005fd0 tx=0x7f98bc009680 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:48.550 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.546+0000 7f98ca7fc640 1 -- 172.21.15.82:0/1203091195 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f98c0076310 con 0x7f98cc101d80 2024-06-06T07:41:48.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.724+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f98cc104a50 con 0x7f98cc101d80 2024-06-06T07:41:48.724 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:48 smithi082 ceph-mon[29354]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:48.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.724+0000 7f98ca7fc640 1 -- 172.21.15.82:0/1203091195 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f98c0004c00 con 0x7f98cc101d80 2024-06-06T07:41:48.725 INFO:teuthology.orchestra.run.smithi082.stdout:111669149717 2024-06-06T07:41:48.726 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f98b406aef0 msgr2=0x7f98b406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f98b406aef0 0x7f98b406d3b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f98bc005fd0 tx=0x7f98bc009680 comp rx=0 tx=0).stop 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 msgr2=0x7f98cc196810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 0x7f98cc196810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f98c0006010 tx=0x7f98c00055a0 comp rx=0 tx=0).stop 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 shutdown_connections 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f98b406aef0 0x7f98b406d3b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98cc101d80 0x7f98cc196810 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 --2- 172.21.15.82:0/1203091195 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98cc101460 0x7f98cc1962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 >> 172.21.15.82:0/1203091195 conn(0x7f98cc075e10 msgr2=0x7f98cc1026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 shutdown_connections 2024-06-06T07:41:48.727 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:48.726+0000 7f98d3687640 1 -- 172.21.15.82:0/1203091195 wait complete. 2024-06-06T07:41:49.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:48 smithi184 ceph-mon[37288]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:49.427 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:49.931 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:49 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1203091195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-06T07:41:49.953 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149715 got 111669149717 for osd.3 2024-06-06T07:41:49.953 DEBUG:teuthology.parallel:result is None 2024-06-06T07:41:50.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:49 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1203091195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-06T07:41:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.449+0000 7ff224673640 1 -- 172.21.15.82:0/899497432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 msgr2=0x7ff21c101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.449+0000 7ff224673640 1 --2- 172.21.15.82:0/899497432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 0x7ff21c101840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff218009880 tx=0x7ff21802a530 comp rx=0 tx=0).stop 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.449+0000 7ff224673640 1 -- 172.21.15.82:0/899497432 shutdown_connections 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.449+0000 7ff224673640 1 --2- 172.21.15.82:0/899497432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff21c101d80 0x7ff21c10a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.449+0000 7ff224673640 1 --2- 172.21.15.82:0/899497432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 0x7ff21c101840 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.449+0000 7ff224673640 1 -- 172.21.15.82:0/899497432 >> 172.21.15.82:0/899497432 conn(0x7ff21c0fac50 msgr2=0x7ff21c0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.449+0000 7ff224673640 1 -- 172.21.15.82:0/899497432 shutdown_connections 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.449+0000 7ff224673640 1 -- 172.21.15.82:0/899497432 wait complete. 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff224673640 1 Processor -- start 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff224673640 1 -- start start 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff224673640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 0x7ff21c196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff224673640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff21c101d80 0x7ff21c196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff224673640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c196ef0 con 0x7ff21c101460 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff224673640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c19ac10 con 0x7ff21c101d80 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff22240f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 0x7ff21c196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff22240f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 0x7ff21c196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:60372/0 (socket says 172.21.15.82:60372) 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff22240f640 1 -- 172.21.15.82:0/781494587 learned_addr learned my addr 172.21.15.82:0/781494587 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff221c0e640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff21c101d80 0x7ff21c196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff22240f640 1 -- 172.21.15.82:0/781494587 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff21c101d80 msgr2=0x7ff21c196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff22240f640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff21c101d80 0x7ff21c196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff22240f640 1 -- 172.21.15.82:0/781494587 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff218009560 con 0x7ff21c101460 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.450+0000 7ff22240f640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 0x7ff21c196320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff21802aa50 tx=0x7ff2180055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.451+0000 7ff2137fe640 1 -- 172.21.15.82:0/781494587 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff218004aa0 con 0x7ff21c101460 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.451+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21c19ae90 con 0x7ff21c101460 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.451+0000 7ff2137fe640 1 -- 172.21.15.82:0/781494587 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff218004c00 con 0x7ff21c101460 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.451+0000 7ff2137fe640 1 -- 172.21.15.82:0/781494587 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff218003ab0 con 0x7ff21c101460 2024-06-06T07:41:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.451+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21c19b380 con 0x7ff21c101460 2024-06-06T07:41:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.452+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21c069020 con 0x7ff21c101460 2024-06-06T07:41:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.452+0000 7ff2137fe640 1 -- 172.21.15.82:0/781494587 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7ff21802ad50 con 0x7ff21c101460 2024-06-06T07:41:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.453+0000 7ff2137fe640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff20806aef0 0x7ff20806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.453+0000 7ff221c0e640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff20806aef0 0x7ff20806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.453+0000 7ff2137fe640 1 -- 172.21.15.82:0/781494587 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff2180aaa50 con 0x7ff21c101460 2024-06-06T07:41:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.453+0000 7ff221c0e640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff20806aef0 0x7ff20806d3b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff21c197880 tx=0x7ff20c008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.457+0000 7ff2137fe640 1 -- 172.21.15.82:0/781494587 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff218076030 con 0x7ff21c101460 2024-06-06T07:41:50.635 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.632+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff21c104aa0 con 0x7ff21c101460 2024-06-06T07:41:50.635 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.633+0000 7ff2137fe640 1 -- 172.21.15.82:0/781494587 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff218079650 con 0x7ff21c101460 2024-06-06T07:41:50.635 INFO:teuthology.orchestra.run.smithi082.stdout:60129542176 2024-06-06T07:41:50.636 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff20806aef0 msgr2=0x7ff20806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:50.636 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff20806aef0 0x7ff20806d3b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff21c197880 tx=0x7ff20c008040 comp rx=0 tx=0).stop 2024-06-06T07:41:50.636 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 msgr2=0x7ff21c196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:50.636 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 0x7ff21c196320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff21802aa50 tx=0x7ff2180055a0 comp rx=0 tx=0).stop 2024-06-06T07:41:50.636 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 shutdown_connections 2024-06-06T07:41:50.636 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff20806aef0 0x7ff20806d3b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:50.636 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff21c101d80 0x7ff21c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:50.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 --2- 172.21.15.82:0/781494587 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff21c101460 0x7ff21c196320 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:50.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 >> 172.21.15.82:0/781494587 conn(0x7ff21c0fac50 msgr2=0x7ff21c102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:50.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 shutdown_connections 2024-06-06T07:41:50.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:50.635+0000 7ff224673640 1 -- 172.21.15.82:0/781494587 wait complete. 2024-06-06T07:41:50.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:50 smithi082 ceph-mon[29354]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:50.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888077 got 146028888080 for osd.4 2024-06-06T07:41:50.891 DEBUG:teuthology.parallel:result is None 2024-06-06T07:41:51.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:50 smithi184 ceph-mon[37288]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:51.562 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-06-06T07:41:51.562 DEBUG:teuthology.parallel:result is None 2024-06-06T07:41:51.750 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.738+0000 7fb85c535640 1 -- 172.21.15.82:0/1993960853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 msgr2=0x7fb854103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:51.750 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.738+0000 7fb85c535640 1 --2- 172.21.15.82:0/1993960853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 0x7fb854103db0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb844009880 tx=0x7fb84402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.738+0000 7fb85c535640 1 -- 172.21.15.82:0/1993960853 shutdown_connections 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.738+0000 7fb85c535640 1 --2- 172.21.15.82:0/1993960853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 0x7fb854103db0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.738+0000 7fb85c535640 1 --2- 172.21.15.82:0/1993960853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8540ff3b0 0x7fb8540ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.738+0000 7fb85c535640 1 -- 172.21.15.82:0/1993960853 >> 172.21.15.82:0/1993960853 conn(0x7fb8540fac80 msgr2=0x7fb8540fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.738+0000 7fb85c535640 1 -- 172.21.15.82:0/1993960853 shutdown_connections 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.738+0000 7fb85c535640 1 -- 172.21.15.82:0/1993960853 wait complete. 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb85c535640 1 Processor -- start 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb85c535640 1 -- start start 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb85c535640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8540ff3b0 0x7fb854196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb85c535640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 0x7fb854196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:51.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb85c535640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb854196f20 con 0x7fb8540ffd60 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb85c535640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85419abf0 con 0x7fb8540ff3b0 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb859ad0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 0x7fb854196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb859ad0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 0x7fb854196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:60396/0 (socket says 172.21.15.82:60396) 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb859ad0640 1 -- 172.21.15.82:0/42397698 learned_addr learned my addr 172.21.15.82:0/42397698 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb859ad0640 1 -- 172.21.15.82:0/42397698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8540ff3b0 msgr2=0x7fb854196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb859ad0640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8540ff3b0 0x7fb854196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb859ad0640 1 -- 172.21.15.82:0/42397698 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb844009560 con 0x7fb8540ffd60 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.739+0000 7fb859ad0640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 0x7fb854196890 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb84402a9c0 tx=0x7fb8440055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.740+0000 7fb84b7fe640 1 -- 172.21.15.82:0/42397698 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb844004aa0 con 0x7fb8540ffd60 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.740+0000 7fb84b7fe640 1 -- 172.21.15.82:0/42397698 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb844004c00 con 0x7fb8540ffd60 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.740+0000 7fb84b7fe640 1 -- 172.21.15.82:0/42397698 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb844003a50 con 0x7fb8540ffd60 2024-06-06T07:41:51.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.740+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85419ae70 con 0x7fb8540ffd60 2024-06-06T07:41:51.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.740+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85419b310 con 0x7fb8540ffd60 2024-06-06T07:41:51.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.741+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb854100890 con 0x7fb8540ffd60 2024-06-06T07:41:51.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.741+0000 7fb84b7fe640 1 -- 172.21.15.82:0/42397698 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb844031c90 con 0x7fb8540ffd60 2024-06-06T07:41:51.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.741+0000 7fb84b7fe640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb84006aef0 0x7fb84006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:51.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.742+0000 7fb84b7fe640 1 -- 172.21.15.82:0/42397698 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb8440aae00 con 0x7fb8540ffd60 2024-06-06T07:41:51.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.742+0000 7fb85a2d1640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb84006aef0 0x7fb84006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:51.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.742+0000 7fb85a2d1640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb84006aef0 0x7fb84006d3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb850009710 tx=0x7fb850006c90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:51.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.746+0000 7fb84b7fe640 1 -- 172.21.15.82:0/42397698 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb8440763e0 con 0x7fb8540ffd60 2024-06-06T07:41:51.912 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:51 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/781494587' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-06T07:41:51.928 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.928+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb85404e210 con 0x7fb8540ffd60 2024-06-06T07:41:51.929 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.929+0000 7fb84b7fe640 1 -- 172.21.15.82:0/42397698 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb844079a00 con 0x7fb8540ffd60 2024-06-06T07:41:51.929 INFO:teuthology.orchestra.run.smithi082.stdout:38654705702 2024-06-06T07:41:51.931 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb84006aef0 msgr2=0x7fb84006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:51.931 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb84006aef0 0x7fb84006d3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb850009710 tx=0x7fb850006c90 comp rx=0 tx=0).stop 2024-06-06T07:41:51.931 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 msgr2=0x7fb854196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:51.931 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 0x7fb854196890 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb84402a9c0 tx=0x7fb8440055a0 comp rx=0 tx=0).stop 2024-06-06T07:41:51.932 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 shutdown_connections 2024-06-06T07:41:51.932 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb84006aef0 0x7fb84006d3b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:51.932 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8540ffd60 0x7fb854196890 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:51.932 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 --2- 172.21.15.82:0/42397698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb8540ff3b0 0x7fb854196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:51.932 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 >> 172.21.15.82:0/42397698 conn(0x7fb8540fac80 msgr2=0x7fb854068990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:51.932 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 shutdown_connections 2024-06-06T07:41:51.932 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:51.931+0000 7fb85c535640 1 -- 172.21.15.82:0/42397698 wait complete. 2024-06-06T07:41:52.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:51 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/781494587' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-06T07:41:52.678 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705702 for osd.0 2024-06-06T07:41:52.679 DEBUG:teuthology.parallel:result is None 2024-06-06T07:41:53.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:52 smithi082 ceph-mon[29354]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:53.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:52 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/42397698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-06T07:41:53.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:41:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:52 smithi184 ceph-mon[37288]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:52 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/42397698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-06T07:41:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:41:54.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.029+0000 7f6090f1e640 1 -- 172.21.15.82:0/2019890556 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ff3a0 msgr2=0x7f608c0ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:54.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.029+0000 7f6090f1e640 1 --2- 172.21.15.82:0/2019890556 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ff3a0 0x7f608c0ff780 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f607c009880 tx=0x7f607c02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:54.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.029+0000 7f6090f1e640 1 -- 172.21.15.82:0/2019890556 shutdown_connections 2024-06-06T07:41:54.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.029+0000 7f6090f1e640 1 --2- 172.21.15.82:0/2019890556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f608c0ffd50 0x7f608c103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:54.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.029+0000 7f6090f1e640 1 --2- 172.21.15.82:0/2019890556 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ff3a0 0x7f608c0ff780 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:54.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.029+0000 7f6090f1e640 1 -- 172.21.15.82:0/2019890556 >> 172.21.15.82:0/2019890556 conn(0x7f608c0fac50 msgr2=0x7f608c0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:54.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.029+0000 7f6090f1e640 1 -- 172.21.15.82:0/2019890556 shutdown_connections 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.029+0000 7f6090f1e640 1 -- 172.21.15.82:0/2019890556 wait complete. 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.030+0000 7f6090f1e640 1 Processor -- start 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.030+0000 7f6090f1e640 1 -- start start 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.030+0000 7f6090f1e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f608c0ff3a0 0x7f608c196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.030+0000 7f6090f1e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ffd50 0x7f608c1967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.030+0000 7f6090f1e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c196e60 con 0x7f608c0ffd50 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.030+0000 7f6090f1e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c19ab80 con 0x7f608c0ff3a0 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.030+0000 7f6089d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ffd50 0x7f608c1967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.030+0000 7f6089d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ffd50 0x7f608c1967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:60418/0 (socket says 172.21.15.82:60418) 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f6089d9b640 1 -- 172.21.15.82:0/3981164461 learned_addr learned my addr 172.21.15.82:0/3981164461 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f608a59c640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f608c0ff3a0 0x7f608c196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f6089d9b640 1 -- 172.21.15.82:0/3981164461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f608c0ff3a0 msgr2=0x7f608c196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:54.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f6089d9b640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f608c0ff3a0 0x7f608c196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f6089d9b640 1 -- 172.21.15.82:0/3981164461 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c009560 con 0x7f608c0ffd50 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f608a59c640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f608c0ff3a0 0x7f608c196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f6089d9b640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ffd50 0x7f608c1967d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f608000d6c0 tx=0x7f608000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f607b7fe640 1 -- 172.21.15.82:0/3981164461 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60800051f0 con 0x7f608c0ffd50 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608c19ae60 con 0x7f608c0ffd50 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f607b7fe640 1 -- 172.21.15.82:0/3981164461 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6080009ce0 con 0x7f608c0ffd50 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f607b7fe640 1 -- 172.21.15.82:0/3981164461 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6080010460 con 0x7f608c0ffd50 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.031+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c19b3b0 con 0x7f608c0ffd50 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.032+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608c100880 con 0x7f608c0ffd50 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.033+0000 7f607b7fe640 1 -- 172.21.15.82:0/3981164461 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f6080005350 con 0x7f608c0ffd50 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.033+0000 7f607b7fe640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f607406af00 0x7f607406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.033+0000 7f607b7fe640 1 -- 172.21.15.82:0/3981164461 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f60800896e0 con 0x7f608c0ffd50 2024-06-06T07:41:54.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.033+0000 7f608a59c640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f607406af00 0x7f607406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:54.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.034+0000 7f608a59c640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f607406af00 0x7f607406d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f607c000c00 tx=0x7f607c0041a0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:54.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.038+0000 7f607b7fe640 1 -- 172.21.15.82:0/3981164461 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6080054cc0 con 0x7f608c0ffd50 2024-06-06T07:41:54.210 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.210+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f608c04e210 con 0x7f608c0ffd50 2024-06-06T07:41:54.210 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.210+0000 7f607b7fe640 1 -- 172.21.15.82:0/3981164461 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f60800582e0 con 0x7f608c0ffd50 2024-06-06T07:41:54.210 INFO:teuthology.orchestra.run.smithi082.stdout:81604378651 2024-06-06T07:41:54.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f607406af00 msgr2=0x7f607406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f607406af00 0x7f607406d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f607c000c00 tx=0x7f607c0041a0 comp rx=0 tx=0).stop 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ffd50 msgr2=0x7f608c1967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ffd50 0x7f608c1967d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f608000d6c0 tx=0x7f608000db80 comp rx=0 tx=0).stop 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 shutdown_connections 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f607406af00 0x7f607406d3c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f608c0ffd50 0x7f608c1967d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 --2- 172.21.15.82:0/3981164461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f608c0ff3a0 0x7f608c196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 >> 172.21.15.82:0/3981164461 conn(0x7f608c0fac50 msgr2=0x7f608c068970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 shutdown_connections 2024-06-06T07:41:54.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:54.212+0000 7f6090f1e640 1 -- 172.21.15.82:0/3981164461 wait complete. 2024-06-06T07:41:54.840 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-06-06T07:41:54.840 DEBUG:teuthology.parallel:result is None 2024-06-06T07:41:54.841 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-06T07:41:54.841 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-06T07:41:55.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:54 smithi082 ceph-mon[29354]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:55.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:54 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3981164461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-06T07:41:55.045 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:55.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:54 smithi184 ceph-mon[37288]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:55.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:54 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3981164461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-06T07:41:56.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.665+0000 7fa801829640 1 -- 172.21.15.82:0/494757309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 msgr2=0x7fa7fc104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:56.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.665+0000 7fa801829640 1 --2- 172.21.15.82:0/494757309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 0x7fa7fc104e00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009880 tx=0x7fa7e402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:56.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.665+0000 7fa801829640 1 -- 172.21.15.82:0/494757309 shutdown_connections 2024-06-06T07:41:56.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.665+0000 7fa801829640 1 --2- 172.21.15.82:0/494757309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 0x7fa7fc104e00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:56.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.665+0000 7fa801829640 1 --2- 172.21.15.82:0/494757309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7fc100400 0x7fa7fc1007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:56.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.665+0000 7fa801829640 1 -- 172.21.15.82:0/494757309 >> 172.21.15.82:0/494757309 conn(0x7fa7fc0fbcb0 msgr2=0x7fa7fc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:56.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.665+0000 7fa801829640 1 -- 172.21.15.82:0/494757309 shutdown_connections 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.665+0000 7fa801829640 1 -- 172.21.15.82:0/494757309 wait complete. 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.665+0000 7fa801829640 1 Processor -- start 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa801829640 1 -- start start 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa801829640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7fc100400 0x7fa7fc196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa801829640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 0x7fa7fc196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa801829640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc196ef0 con 0x7fa7fc100db0 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa801829640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc19ac10 con 0x7fa7fc100400 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa7f3fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 0x7fa7fc196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa7f3fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 0x7fa7fc196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:44158/0 (socket says 172.21.15.82:44158) 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa7f3fff640 1 -- 172.21.15.82:0/2784895670 learned_addr learned my addr 172.21.15.82:0/2784895670 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:56.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa7faffd640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7fc100400 0x7fa7fc196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa7f3fff640 1 -- 172.21.15.82:0/2784895670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7fc100400 msgr2=0x7fa7fc196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa7f3fff640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7fc100400 0x7fa7fc196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa7f3fff640 1 -- 172.21.15.82:0/2784895670 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e4009560 con 0x7fa7fc100db0 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa7faffd640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7fc100400 0x7fa7fc196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa7f3fff640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 0x7fa7fc196860 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa7e402a9c0 tx=0x7fa7e40055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.666+0000 7fa800827640 1 -- 172.21.15.82:0/2784895670 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7e4004aa0 con 0x7fa7fc100db0 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.667+0000 7fa800827640 1 -- 172.21.15.82:0/2784895670 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7e4004c00 con 0x7fa7fc100db0 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.667+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc19ae90 con 0x7fa7fc100db0 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.667+0000 7fa800827640 1 -- 172.21.15.82:0/2784895670 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7e4003a50 con 0x7fa7fc100db0 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.667+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc19b380 con 0x7fa7fc100db0 2024-06-06T07:41:56.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.668+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc1018e0 con 0x7fa7fc100db0 2024-06-06T07:41:56.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.668+0000 7fa800827640 1 -- 172.21.15.82:0/2784895670 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fa7e4031c90 con 0x7fa7fc100db0 2024-06-06T07:41:56.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.668+0000 7fa800827640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa7e806aef0 0x7fa7e806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:56.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.668+0000 7fa800827640 1 -- 172.21.15.82:0/2784895670 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa7e40aad20 con 0x7fa7fc100db0 2024-06-06T07:41:56.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.668+0000 7fa7faffd640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa7e806aef0 0x7fa7e806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:56.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.669+0000 7fa7faffd640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa7e806aef0 0x7fa7e806d3b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec006fd0 tx=0x7fa7ec008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:56.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.673+0000 7fa800827640 1 -- 172.21.15.82:0/2784895670 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa7e4076380 con 0x7fa7fc100db0 2024-06-06T07:41:56.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:56 smithi082 ceph-mon[29354]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:56.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.844+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 --> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7fc10d490 con 0x7fa7e806aef0 2024-06-06T07:41:56.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.845+0000 7fa800827640 1 -- 172.21.15.82:0/2784895670 <== mgr.14221 v2:172.21.15.82:6800/2676832140 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28002 (secure 0 0 0) 0x7fa7fc10d490 con 0x7fa7e806aef0 2024-06-06T07:41:56.846 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:41:56.846 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-06-06T07:41:56.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.847+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa7e806aef0 msgr2=0x7fa7e806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:56.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.847+0000 7fa801829640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa7e806aef0 0x7fa7e806d3b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec006fd0 tx=0x7fa7ec008040 comp rx=0 tx=0).stop 2024-06-06T07:41:56.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.848+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 msgr2=0x7fa7fc196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:56.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.848+0000 7fa801829640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 0x7fa7fc196860 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa7e402a9c0 tx=0x7fa7e40055a0 comp rx=0 tx=0).stop 2024-06-06T07:41:56.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.848+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 shutdown_connections 2024-06-06T07:41:56.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.848+0000 7fa801829640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa7e806aef0 0x7fa7e806d3b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:56.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.848+0000 7fa801829640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7fc100db0 0x7fa7fc196860 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:56.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.848+0000 7fa801829640 1 --2- 172.21.15.82:0/2784895670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa7fc100400 0x7fa7fc196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:56.849 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.848+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 >> 172.21.15.82:0/2784895670 conn(0x7fa7fc0fbcb0 msgr2=0x7fa7fc0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:56.849 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.848+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 shutdown_connections 2024-06-06T07:41:56.849 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:56.848+0000 7fa801829640 1 -- 172.21.15.82:0/2784895670 wait complete. 2024-06-06T07:41:57.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:56 smithi184 ceph-mon[37288]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:57.555 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-06-06T07:41:55.570332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":178292,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":562292620,"statfs":{"total":575970213888,"available":575787642880,"internally_reserved":0,"allocated":3969024,"data_stored":2606850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002608"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-06T07:41:04.558672+0000","last_change":"2024-06-06T07:40:39.646919+0000","last_active":"2024-06-06T07:41:04.558672+0000","last_peered":"2024-06-06T07:41:04.558672+0000","last_clean":"2024-06-06T07:41:04.558672+0000","last_became_active":"2024-06-06T07:40:39.646655+0000","last_became_peered":"2024-06-06T07:40:39.646655+0000","last_unstale":"2024-06-06T07:41:04.558672+0000","last_undegraded":"2024-06-06T07:41:04.558672+0000","last_fullsized":"2024-06-06T07:41:04.558672+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-06T07:39:55.913306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-06T07:39:55.913306+0000","last_clean_scrub_stamp":"2024-06-06T07:39:55.913306+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-07T17:46:02.136207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31472,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713680,"statfs":{"total":95995035648,"available":95962808320,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.5},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 07:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.39800000000000002}]},{"osd":1,"last update":"Thu Jun 6 07:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.443},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jun 6 07:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jun 6 07:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.308},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.41099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31472,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713680,"statfs":{"total":95995035648,"available":95962808320,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 07:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jun 6 07:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jun 6 07:41:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jun 6 07:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 07:41:28 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.316,"15min":0.318},"min":{"1min":0.217,"5min":0.182,"15min":0.182},"max":{"1min":0.39100000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.315},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.32800000000000001,"15min":0.33000000000000002},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45400000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jun 6 07:40:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.36099999999999999}]},{"osd":3,"last update":"Thu Jun 6 07:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.5},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.49099999999999999}]},{"osd":4,"last update":"Thu Jun 6 07:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 6 07:41:32 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.317,"15min":0.315},"min":{"1min":0.253,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.44900000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.32200000000000001,"15min":0.32000000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45900000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.32100000000000001}]},{"osd":2,"last update":"Thu Jun 6 07:40:59 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.29099999999999998}]},{"osd":3,"last update":"Thu Jun 6 07:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.64400000000000002}]},{"osd":4,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":7.4969999999999999,"5min":7.4969999999999999,"15min":7.4969999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-06T07:41:57.556 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-06T07:41:57.764 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:41:59.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:58 smithi082 ceph-mon[29354]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T07:41:59.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:41:58 smithi082 ceph-mon[29354]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:59.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:58 smithi184 ceph-mon[37288]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T07:41:59.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:41:58 smithi184 ceph-mon[37288]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:41:59.558 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 -- 172.21.15.82:0/1414459619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ffd30 msgr2=0x7f6640103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:59.558 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 --2- 172.21.15.82:0/1414459619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ffd30 0x7f6640103d80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6628009880 tx=0x7f662802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:41:59.558 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 -- 172.21.15.82:0/1414459619 shutdown_connections 2024-06-06T07:41:59.558 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 --2- 172.21.15.82:0/1414459619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ffd30 0x7f6640103d80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:59.558 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 --2- 172.21.15.82:0/1414459619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66400ff380 0x7f66400ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:59.558 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 -- 172.21.15.82:0/1414459619 >> 172.21.15.82:0/1414459619 conn(0x7f66400fac30 msgr2=0x7f66400fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:59.558 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 -- 172.21.15.82:0/1414459619 shutdown_connections 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 -- 172.21.15.82:0/1414459619 wait complete. 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 Processor -- start 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 -- start start 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ff380 0x7f66401952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66400ffd30 0x7f6640195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640195eb0 con 0x7f66400ff380 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.553+0000 7f664601b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640199bd0 con 0x7f66400ffd30 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663f7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ff380 0x7f66401952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663f7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ff380 0x7f66401952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:44182/0 (socket says 172.21.15.82:44182) 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663f7fe640 1 -- 172.21.15.82:0/1219522531 learned_addr learned my addr 172.21.15.82:0/1219522531 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663f7fe640 1 -- 172.21.15.82:0/1219522531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66400ffd30 msgr2=0x7f6640195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f6637fff640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66400ffd30 0x7f6640195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663f7fe640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66400ffd30 0x7f6640195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663f7fe640 1 -- 172.21.15.82:0/1219522531 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6628009560 con 0x7f66400ff380 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663f7fe640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ff380 0x7f66401952e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f663000b420 tx=0x7f663000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663d7fa640 1 -- 172.21.15.82:0/1219522531 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6630011800 con 0x7f66400ff380 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663d7fa640 1 -- 172.21.15.82:0/1219522531 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6630011e20 con 0x7f66400ff380 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6640199eb0 con 0x7f66400ff380 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f663d7fa640 1 -- 172.21.15.82:0/1219522531 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663000f2e0 con 0x7f66400ff380 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.554+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664019a400 con 0x7f66400ff380 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.555+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6640100860 con 0x7f66400ff380 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.556+0000 7f663d7fa640 1 -- 172.21.15.82:0/1219522531 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f6630010380 con 0x7f66400ff380 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.556+0000 7f663d7fa640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f662c06aef0 0x7f662c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.556+0000 7f663d7fa640 1 -- 172.21.15.82:0/1219522531 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6630089570 con 0x7f66400ff380 2024-06-06T07:41:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.556+0000 7f6637fff640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f662c06aef0 0x7f662c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:41:59.561 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.556+0000 7f6637fff640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f662c06aef0 0x7f662c06d3b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6628000c00 tx=0x7f6628004120 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:41:59.561 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.560+0000 7f663d7fa640 1 -- 172.21.15.82:0/1219522531 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6630054b50 con 0x7f66400ff380 2024-06-06T07:41:59.730 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.729+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 --> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66401965f0 con 0x7f662c06aef0 2024-06-06T07:41:59.731 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.730+0000 7f663d7fa640 1 -- 172.21.15.82:0/1219522531 <== mgr.14221 v2:172.21.15.82:6800/2676832140 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28018 (secure 0 0 0) 0x7f66401965f0 con 0x7f662c06aef0 2024-06-06T07:41:59.732 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:41:59.732 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-06-06T07:41:59.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.732+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f662c06aef0 msgr2=0x7f662c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.732+0000 7f664601b640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f662c06aef0 0x7f662c06d3b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6628000c00 tx=0x7f6628004120 comp rx=0 tx=0).stop 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.732+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ff380 msgr2=0x7f66401952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.732+0000 7f664601b640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ff380 0x7f66401952e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f663000b420 tx=0x7f663000b8e0 comp rx=0 tx=0).stop 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.732+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 shutdown_connections 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.733+0000 7f664601b640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f662c06aef0 0x7f662c06d3b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.733+0000 7f664601b640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f66400ffd30 0x7f6640195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.733+0000 7f664601b640 1 --2- 172.21.15.82:0/1219522531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66400ff380 0x7f66401952e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.733+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 >> 172.21.15.82:0/1219522531 conn(0x7f66400fac30 msgr2=0x7f66400fbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.733+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 shutdown_connections 2024-06-06T07:41:59.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:41:59.733+0000 7f664601b640 1 -- 172.21.15.82:0/1219522531 wait complete. 2024-06-06T07:42:00.359 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-06-06T07:41:59.570975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":178292,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":174326,"kb_avail":562292620,"statfs":{"total":575970213888,"available":575787642880,"internally_reserved":0,"allocated":3969024,"data_stored":2606850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":178510503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002424"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-06T07:41:04.558672+0000","last_change":"2024-06-06T07:40:39.646919+0000","last_active":"2024-06-06T07:41:04.558672+0000","last_peered":"2024-06-06T07:41:04.558672+0000","last_clean":"2024-06-06T07:41:04.558672+0000","last_became_active":"2024-06-06T07:40:39.646655+0000","last_became_peered":"2024-06-06T07:40:39.646655+0000","last_unstale":"2024-06-06T07:41:04.558672+0000","last_undegraded":"2024-06-06T07:41:04.558672+0000","last_fullsized":"2024-06-06T07:41:04.558672+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-06T07:39:55.913306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-06T07:39:55.913306+0000","last_clean_scrub_stamp":"2024-06-06T07:39:55.913306+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-07T17:46:02.136207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31472,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713680,"statfs":{"total":95995035648,"available":95962808320,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.41999999999999998}]},{"osd":1,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.437},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.32500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 07:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jun 6 07:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.309},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.37}]},{"osd":2,"last update":"Thu Jun 6 07:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.438},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jun 6 07:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.41999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31472,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713680,"statfs":{"total":95995035648,"available":95962808320,"internally_reserved":0,"allocated":364544,"data_stored":139291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 07:41:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36799999999999999,"15min":0.37},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.47299999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36399999999999999,"15min":0.36599999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52900000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.23699999999999999}]},{"osd":1,"last update":"Thu Jun 6 07:41:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.214,"15min":0.214},"max":{"1min":0.49299999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39500000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.29499999999999998}]},{"osd":3,"last update":"Thu Jun 6 07:41:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.375},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jun 6 07:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.35899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 07:41:28 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.316,"15min":0.318},"min":{"1min":0.217,"5min":0.182,"15min":0.182},"max":{"1min":0.39100000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.32800000000000001,"15min":0.33000000000000002},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45400000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.28699999999999998}]},{"osd":2,"last update":"Thu Jun 6 07:40:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.30399999999999999}]},{"osd":3,"last update":"Thu Jun 6 07:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jun 6 07:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":958464,"data_stored":729659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 6 07:41:32 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.317,"15min":0.315},"min":{"1min":0.253,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.44900000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.32200000000000001,"15min":0.32000000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.45900000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.28899999999999998}]},{"osd":2,"last update":"Thu Jun 6 07:40:59 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.246}]},{"osd":3,"last update":"Thu Jun 6 07:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.45700000000000002}]},{"osd":4,"last update":"Thu Jun 6 07:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":7.4969999999999999,"5min":7.4969999999999999,"15min":7.4969999999999999},"last":0.503},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-06T07:42:00.360 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-06T07:42:00.360 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-06T07:42:00.360 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-06T07:42:00.360 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:00.563 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:00.900 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:00 smithi082 ceph-mon[29354]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:00.900 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:00 smithi082 ceph-mon[29354]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T07:42:01.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:00 smithi184 ceph-mon[37288]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:01.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:00 smithi184 ceph-mon[37288]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T07:42:02.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 -- 172.21.15.82:0/3126937745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ffd10 msgr2=0x7fdc38103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:02.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 --2- 172.21.15.82:0/3126937745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ffd10 0x7fdc38103d60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009880 tx=0x7fdc2002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:42:02.159 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 -- 172.21.15.82:0/3126937745 shutdown_connections 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 --2- 172.21.15.82:0/3126937745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ffd10 0x7fdc38103d60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 --2- 172.21.15.82:0/3126937745 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc380ff360 0x7fdc380ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 -- 172.21.15.82:0/3126937745 >> 172.21.15.82:0/3126937745 conn(0x7fdc38075e10 msgr2=0x7fdc380fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 -- 172.21.15.82:0/3126937745 shutdown_connections 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 -- 172.21.15.82:0/3126937745 wait complete. 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 Processor -- start 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.156+0000 7fdc3e175640 1 -- start start 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc3e175640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ff360 0x7fdc381952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc3e175640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc380ffd10 0x7fdc38195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc3e175640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38195e90 con 0x7fdc380ff360 2024-06-06T07:42:02.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc3e175640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38199bb0 con 0x7fdc380ffd10 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc377fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ff360 0x7fdc381952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc377fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ff360 0x7fdc381952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:44204/0 (socket says 172.21.15.82:44204) 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc377fe640 1 -- 172.21.15.82:0/11378229 learned_addr learned my addr 172.21.15.82:0/11378229 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc2ffff640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc380ffd10 0x7fdc38195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc377fe640 1 -- 172.21.15.82:0/11378229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc380ffd10 msgr2=0x7fdc38195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc377fe640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc380ffd10 0x7fdc38195800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc377fe640 1 -- 172.21.15.82:0/11378229 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc20009560 con 0x7fdc380ff360 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc377fe640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ff360 0x7fdc381952c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800b420 tx=0x7fdc2800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc357fa640 1 -- 172.21.15.82:0/11378229 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc28011800 con 0x7fdc380ff360 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc357fa640 1 -- 172.21.15.82:0/11378229 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc28011e20 con 0x7fdc380ff360 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc38199e90 con 0x7fdc380ff360 2024-06-06T07:42:02.161 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.157+0000 7fdc357fa640 1 -- 172.21.15.82:0/11378229 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc2800f2e0 con 0x7fdc380ff360 2024-06-06T07:42:02.162 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.158+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3819a3b0 con 0x7fdc380ff360 2024-06-06T07:42:02.162 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.158+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc38100840 con 0x7fdc380ff360 2024-06-06T07:42:02.162 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.159+0000 7fdc357fa640 1 -- 172.21.15.82:0/11378229 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fdc28010380 con 0x7fdc380ff360 2024-06-06T07:42:02.162 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.159+0000 7fdc357fa640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fdc2406aef0 0x7fdc2406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:02.162 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.160+0000 7fdc357fa640 1 -- 172.21.15.82:0/11378229 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdc280894d0 con 0x7fdc380ff360 2024-06-06T07:42:02.163 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.160+0000 7fdc2ffff640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fdc2406aef0 0x7fdc2406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:02.163 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.160+0000 7fdc2ffff640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fdc2406aef0 0x7fdc2406d3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009230 tx=0x7fdc20005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:02.164 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.164+0000 7fdc357fa640 1 -- 172.21.15.82:0/11378229 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fdc28054b30 con 0x7fdc380ff360 2024-06-06T07:42:02.375 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.374+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdc381965d0 con 0x7fdc380ff360 2024-06-06T07:42:02.375 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.374+0000 7fdc357fa640 1 -- 172.21.15.82:0/11378229 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdc28058150 con 0x7fdc380ff360 2024-06-06T07:42:02.375 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:02.375 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:02.377 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fdc2406aef0 msgr2=0x7fdc2406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:02.377 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fdc2406aef0 0x7fdc2406d3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009230 tx=0x7fdc20005a20 comp rx=0 tx=0).stop 2024-06-06T07:42:02.377 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ff360 msgr2=0x7fdc381952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:02.377 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ff360 0x7fdc381952c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800b420 tx=0x7fdc2800b8e0 comp rx=0 tx=0).stop 2024-06-06T07:42:02.377 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 shutdown_connections 2024-06-06T07:42:02.378 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fdc2406aef0 0x7fdc2406d3b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:02.378 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc380ffd10 0x7fdc38195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:02.378 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 --2- 172.21.15.82:0/11378229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdc380ff360 0x7fdc381952c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:02.378 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 >> 172.21.15.82:0/11378229 conn(0x7fdc38075e10 msgr2=0x7fdc38102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:02.378 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 shutdown_connections 2024-06-06T07:42:02.378 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:02.376+0000 7fdc3e175640 1 -- 172.21.15.82:0/11378229 wait complete. 2024-06-06T07:42:03.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:02 smithi082 ceph-mon[29354]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:03.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:02 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/11378229' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:03.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:02 smithi184 ceph-mon[37288]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:03.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:02 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/11378229' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:04.912 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:04 smithi184 ceph-mon[37288]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:05.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:04 smithi082 ceph-mon[29354]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:06.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:05 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:42:06.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:05 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:42:06.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:05 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:42:06.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:05 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:42:06.073 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:06.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:05 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:42:06.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:05 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:42:06.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:05 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:42:06.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:05 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:42:06.283 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:07.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:06 smithi082 ceph-mon[29354]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:06 smithi184 ceph-mon[37288]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:07.933 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:42:07.938 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.936+0000 7f0f7beeb640 1 -- 172.21.15.82:0/1903874025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 msgr2=0x7f0f7410a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:07.938 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.936+0000 7f0f7beeb640 1 --2- 172.21.15.82:0/1903874025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 0x7f0f7410a250 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009880 tx=0x7f0f6402a530 comp rx=0 tx=0).stop 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.936+0000 7f0f7beeb640 1 -- 172.21.15.82:0/1903874025 shutdown_connections 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.936+0000 7f0f7beeb640 1 --2- 172.21.15.82:0/1903874025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 0x7f0f7410a250 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.936+0000 7f0f7beeb640 1 --2- 172.21.15.82:0/1903874025 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f74101440 0x7f0f74101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.936+0000 7f0f7beeb640 1 -- 172.21.15.82:0/1903874025 >> 172.21.15.82:0/1903874025 conn(0x7f0f74075e10 msgr2=0x7f0f740fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.937+0000 7f0f7beeb640 1 -- 172.21.15.82:0/1903874025 shutdown_connections 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.937+0000 7f0f7beeb640 1 -- 172.21.15.82:0/1903874025 wait complete. 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.937+0000 7f0f7beeb640 1 Processor -- start 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.937+0000 7f0f7beeb640 1 -- start start 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.937+0000 7f0f7beeb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f74101440 0x7f0f741961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.937+0000 7f0f7beeb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 0x7f0f74196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:07.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.937+0000 7f0f7beeb640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f74196db0 con 0x7f0f74101d60 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.937+0000 7f0f7beeb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7419aad0 con 0x7f0f74101440 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.937+0000 7f0f79486640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 0x7f0f74196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f79486640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 0x7f0f74196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:38960/0 (socket says 172.21.15.82:38960) 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f79486640 1 -- 172.21.15.82:0/2508753412 learned_addr learned my addr 172.21.15.82:0/2508753412 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f79c87640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f74101440 0x7f0f741961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f79486640 1 -- 172.21.15.82:0/2508753412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f74101440 msgr2=0x7f0f741961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f79486640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f74101440 0x7f0f741961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f79486640 1 -- 172.21.15.82:0/2508753412 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f64009560 con 0x7f0f74101d60 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f79486640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 0x7f0f74196720 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0f6402aa50 tx=0x7f0f640055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f6affd640 1 -- 172.21.15.82:0/2508753412 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f64004aa0 con 0x7f0f74101d60 2024-06-06T07:42:07.940 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f6affd640 1 -- 172.21.15.82:0/2508753412 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f64004c00 con 0x7f0f74101d60 2024-06-06T07:42:07.941 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7419ad50 con 0x7f0f74101d60 2024-06-06T07:42:07.941 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f6affd640 1 -- 172.21.15.82:0/2508753412 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f64003980 con 0x7f0f74101d60 2024-06-06T07:42:07.941 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.938+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7419b240 con 0x7f0f74101d60 2024-06-06T07:42:07.941 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.939+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f74068490 con 0x7f0f74101d60 2024-06-06T07:42:07.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.940+0000 7f0f6affd640 1 -- 172.21.15.82:0/2508753412 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0f6402ad50 con 0x7f0f74101d60 2024-06-06T07:42:07.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.940+0000 7f0f6affd640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f6006aef0 0x7f0f6006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:07.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.940+0000 7f0f6affd640 1 -- 172.21.15.82:0/2508753412 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0f640aa950 con 0x7f0f74101d60 2024-06-06T07:42:07.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.940+0000 7f0f79c87640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f6006aef0 0x7f0f6006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:07.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.940+0000 7f0f79c87640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f6006aef0 0x7f0f6006d3b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000bbe0 tx=0x7f0f7000b390 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:07.944 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:07.944+0000 7f0f6affd640 1 -- 172.21.15.82:0/2508753412 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0f64075fe0 con 0x7f0f74101d60 2024-06-06T07:42:08.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:42:08.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.152+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0f7404e210 con 0x7f0f74101d60 2024-06-06T07:42:08.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.153+0000 7f0f6affd640 1 -- 172.21.15.82:0/2508753412 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0f64079600 con 0x7f0f74101d60 2024-06-06T07:42:08.153 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:08.153 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:08.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.154+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f6006aef0 msgr2=0x7f0f6006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:08.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.154+0000 7f0f7beeb640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f6006aef0 0x7f0f6006d3b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000bbe0 tx=0x7f0f7000b390 comp rx=0 tx=0).stop 2024-06-06T07:42:08.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.155+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 msgr2=0x7f0f74196720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:08.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.155+0000 7f0f7beeb640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 0x7f0f74196720 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0f6402aa50 tx=0x7f0f640055a0 comp rx=0 tx=0).stop 2024-06-06T07:42:08.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.155+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 shutdown_connections 2024-06-06T07:42:08.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.155+0000 7f0f7beeb640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f6006aef0 0x7f0f6006d3b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:08.156 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.155+0000 7f0f7beeb640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f74101d60 0x7f0f74196720 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:08.156 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.155+0000 7f0f7beeb640 1 --2- 172.21.15.82:0/2508753412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f74101440 0x7f0f741961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:08.156 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.155+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 >> 172.21.15.82:0/2508753412 conn(0x7f0f74075e10 msgr2=0x7f0f741026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:08.156 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.155+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 shutdown_connections 2024-06-06T07:42:08.156 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:08.155+0000 7f0f7beeb640 1 -- 172.21.15.82:0/2508753412 wait complete. 2024-06-06T07:42:08.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:08 smithi082 ceph-mon[29354]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:08.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:08 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2508753412' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:09.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:08 smithi184 ceph-mon[37288]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:09.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:08 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2508753412' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:11.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:10 smithi082 ceph-mon[29354]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:11.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:10 smithi184 ceph-mon[37288]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:11.854 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:12.072 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:12.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:12 smithi082 ceph-mon[29354]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:13.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:12 smithi184 ceph-mon[37288]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:13.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.769+0000 7fcafbb37640 1 -- 172.21.15.82:0/482904808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 msgr2=0x7fcaf4104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:13.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.769+0000 7fcafbb37640 1 --2- 172.21.15.82:0/482904808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 0x7fcaf4104eb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009880 tx=0x7fcae402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:42:13.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.769+0000 7fcafbb37640 1 -- 172.21.15.82:0/482904808 shutdown_connections 2024-06-06T07:42:13.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.769+0000 7fcafbb37640 1 --2- 172.21.15.82:0/482904808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 0x7fcaf4104eb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:13.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.769+0000 7fcafbb37640 1 --2- 172.21.15.82:0/482904808 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaf41004b0 0x7fcaf4100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:13.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.769+0000 7fcafbb37640 1 -- 172.21.15.82:0/482904808 >> 172.21.15.82:0/482904808 conn(0x7fcaf40fbda0 msgr2=0x7fcaf40fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:13.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.769+0000 7fcafbb37640 1 -- 172.21.15.82:0/482904808 shutdown_connections 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.769+0000 7fcafbb37640 1 -- 172.21.15.82:0/482904808 wait complete. 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.770+0000 7fcafbb37640 1 Processor -- start 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.770+0000 7fcafbb37640 1 -- start start 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.770+0000 7fcafbb37640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaf41004b0 0x7fcaf41963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.770+0000 7fcafbb37640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 0x7fcaf4196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.770+0000 7fcafbb37640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf4196fa0 con 0x7fcaf4100e60 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.770+0000 7fcafbb37640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf419acc0 con 0x7fcaf41004b0 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.770+0000 7fcaf90d2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 0x7fcaf4196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.770+0000 7fcaf90d2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 0x7fcaf4196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:38982/0 (socket says 172.21.15.82:38982) 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.770+0000 7fcaf90d2640 1 -- 172.21.15.82:0/3708086045 learned_addr learned my addr 172.21.15.82:0/3708086045 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcaf98d3640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaf41004b0 0x7fcaf41963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:13.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcaf90d2640 1 -- 172.21.15.82:0/3708086045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaf41004b0 msgr2=0x7fcaf41963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcaf90d2640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaf41004b0 0x7fcaf41963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcaf90d2640 1 -- 172.21.15.82:0/3708086045 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae4009560 con 0x7fcaf4100e60 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcaf98d3640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaf41004b0 0x7fcaf41963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcaf90d2640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 0x7fcaf4196910 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcae402a9c0 tx=0x7fcae40055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcaeaffd640 1 -- 172.21.15.82:0/3708086045 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcae4004aa0 con 0x7fcaf4100e60 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcaeaffd640 1 -- 172.21.15.82:0/3708086045 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcae4004c00 con 0x7fcaf4100e60 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf419af40 con 0x7fcaf4100e60 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcaeaffd640 1 -- 172.21.15.82:0/3708086045 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcae4003a50 con 0x7fcaf4100e60 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.771+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf419b430 con 0x7fcaf4100e60 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.772+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf4101990 con 0x7fcaf4100e60 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.772+0000 7fcaeaffd640 1 -- 172.21.15.82:0/3708086045 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fcae4031c90 con 0x7fcaf4100e60 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.773+0000 7fcaeaffd640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcae006aef0 0x7fcae006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.773+0000 7fcaeaffd640 1 -- 172.21.15.82:0/3708086045 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fcae40aae20 con 0x7fcaf4100e60 2024-06-06T07:42:13.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.773+0000 7fcaf98d3640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcae006aef0 0x7fcae006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:13.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.773+0000 7fcaf98d3640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcae006aef0 0x7fcae006d3b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0006fd0 tx=0x7fcaf0008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:13.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.777+0000 7fcaeaffd640 1 -- 172.21.15.82:0/3708086045 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fcae40763a0 con 0x7fcaf4100e60 2024-06-06T07:42:13.990 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.990+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcaf404e210 con 0x7fcaf4100e60 2024-06-06T07:42:13.990 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.990+0000 7fcaeaffd640 1 -- 172.21.15.82:0/3708086045 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcae40799c0 con 0x7fcaf4100e60 2024-06-06T07:42:13.990 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:13.990 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:13.992 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcae006aef0 msgr2=0x7fcae006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:13.992 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcae006aef0 0x7fcae006d3b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0006fd0 tx=0x7fcaf0008040 comp rx=0 tx=0).stop 2024-06-06T07:42:13.992 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 msgr2=0x7fcaf4196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:13.992 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 0x7fcaf4196910 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcae402a9c0 tx=0x7fcae40055a0 comp rx=0 tx=0).stop 2024-06-06T07:42:13.992 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 shutdown_connections 2024-06-06T07:42:13.992 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcae006aef0 0x7fcae006d3b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:13.992 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaf4100e60 0x7fcaf4196910 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:13.993 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 --2- 172.21.15.82:0/3708086045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcaf41004b0 0x7fcaf41963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:13.993 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 >> 172.21.15.82:0/3708086045 conn(0x7fcaf40fbda0 msgr2=0x7fcaf40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:13.993 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 shutdown_connections 2024-06-06T07:42:13.993 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:13.992+0000 7fcafbb37640 1 -- 172.21.15.82:0/3708086045 wait complete. 2024-06-06T07:42:15.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:14 smithi082 ceph-mon[29354]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:15.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:14 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3708086045' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:15.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:14 smithi184 ceph-mon[37288]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:15.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:14 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3708086045' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:17.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:16 smithi082 ceph-mon[29354]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:17.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:16 smithi184 ceph-mon[37288]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:17.582 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:17.802 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:18.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:18 smithi082 ceph-mon[29354]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:18 smithi184 ceph-mon[37288]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:19.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.346+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2126066861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 msgr2=0x7fc7e0101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:19.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.346+0000 7fc7e49f2640 1 --2- 172.21.15.82:0/2126066861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 0x7fc7e0101840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40098b0 tx=0x7fc7d402a560 comp rx=0 tx=0).stop 2024-06-06T07:42:19.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.346+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2126066861 shutdown_connections 2024-06-06T07:42:19.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.346+0000 7fc7e49f2640 1 --2- 172.21.15.82:0/2126066861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7e0101d80 0x7fc7e010a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:19.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.346+0000 7fc7e49f2640 1 --2- 172.21.15.82:0/2126066861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 0x7fc7e0101840 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:19.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.346+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2126066861 >> 172.21.15.82:0/2126066861 conn(0x7fc7e0075e10 msgr2=0x7fc7e00fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:19.393 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.346+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2126066861 shutdown_connections 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.346+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2126066861 wait complete. 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7e49f2640 1 Processor -- start 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7e49f2640 1 -- start start 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7e49f2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 0x7fc7e0196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7e49f2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7e0101d80 0x7fc7e0196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7e49f2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e0196f00 con 0x7fc7e0101460 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7e49f2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e019ac20 con 0x7fc7e0101d80 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7de59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 0x7fc7e0196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7de59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 0x7fc7e0196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:39388/0 (socket says 172.21.15.82:39388) 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7de59c640 1 -- 172.21.15.82:0/2009214509 learned_addr learned my addr 172.21.15.82:0/2009214509 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7de59c640 1 -- 172.21.15.82:0/2009214509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7e0101d80 msgr2=0x7fc7e0196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:19.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7de59c640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7e0101d80 0x7fc7e0196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.347+0000 7fc7de59c640 1 -- 172.21.15.82:0/2009214509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4009560 con 0x7fc7e0101460 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.348+0000 7fc7de59c640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 0x7fc7e0196330 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4006010 tx=0x7fc7d40055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.348+0000 7fc7cf7fe640 1 -- 172.21.15.82:0/2009214509 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7d4004aa0 con 0x7fc7e0101460 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.348+0000 7fc7cf7fe640 1 -- 172.21.15.82:0/2009214509 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7d4004c00 con 0x7fc7e0101460 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.348+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e019aea0 con 0x7fc7e0101460 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.348+0000 7fc7cf7fe640 1 -- 172.21.15.82:0/2009214509 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7d4003a20 con 0x7fc7e0101460 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.348+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e019b390 con 0x7fc7e0101460 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.349+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e0068470 con 0x7fc7e0101460 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.349+0000 7fc7cf7fe640 1 -- 172.21.15.82:0/2009214509 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fc7d4003c90 con 0x7fc7e0101460 2024-06-06T07:42:19.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.350+0000 7fc7cf7fe640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc7c806aef0 0x7fc7c806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:19.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.350+0000 7fc7ddd9b640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc7c806aef0 0x7fc7c806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:19.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.350+0000 7fc7cf7fe640 1 -- 172.21.15.82:0/2009214509 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc7d40aadd0 con 0x7fc7e0101460 2024-06-06T07:42:19.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.350+0000 7fc7ddd9b640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc7c806aef0 0x7fc7c806d3b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0197890 tx=0x7fc7d0009250 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:19.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.354+0000 7fc7cf7fe640 1 -- 172.21.15.82:0/2009214509 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc7d40763b0 con 0x7fc7e0101460 2024-06-06T07:42:19.564 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.564+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc7e004e210 con 0x7fc7e0101460 2024-06-06T07:42:19.565 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.565+0000 7fc7cf7fe640 1 -- 172.21.15.82:0/2009214509 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc7d40799d0 con 0x7fc7e0101460 2024-06-06T07:42:19.565 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:19.565 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:19.567 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.566+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc7c806aef0 msgr2=0x7fc7c806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:19.567 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.566+0000 7fc7e49f2640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc7c806aef0 0x7fc7c806d3b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0197890 tx=0x7fc7d0009250 comp rx=0 tx=0).stop 2024-06-06T07:42:19.567 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.567+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 msgr2=0x7fc7e0196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:19.567 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.567+0000 7fc7e49f2640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 0x7fc7e0196330 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4006010 tx=0x7fc7d40055a0 comp rx=0 tx=0).stop 2024-06-06T07:42:19.567 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.567+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 shutdown_connections 2024-06-06T07:42:19.567 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.567+0000 7fc7e49f2640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc7c806aef0 0x7fc7c806d3b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:19.568 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.567+0000 7fc7e49f2640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7e0101d80 0x7fc7e0196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:19.568 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.567+0000 7fc7e49f2640 1 --2- 172.21.15.82:0/2009214509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7e0101460 0x7fc7e0196330 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:19.568 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.567+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 >> 172.21.15.82:0/2009214509 conn(0x7fc7e0075e10 msgr2=0x7fc7e0102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:19.568 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.567+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 shutdown_connections 2024-06-06T07:42:19.568 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:19.567+0000 7fc7e49f2640 1 -- 172.21.15.82:0/2009214509 wait complete. 2024-06-06T07:42:19.937 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:19 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2009214509' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:20.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:19 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2009214509' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:21.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:20 smithi082 ceph-mon[29354]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:21.034 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:20 smithi184 ceph-mon[37288]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:23.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:22 smithi082 ceph-mon[29354]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:23.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:42:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:22 smithi184 ceph-mon[37288]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:42:23.218 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:23.434 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:24.807 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:24 smithi082 ceph-mon[29354]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:25.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.993+0000 7fd45f357640 1 -- 172.21.15.82:0/940536446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ff360 msgr2=0x7fd4580ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:25.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.993+0000 7fd45f357640 1 --2- 172.21.15.82:0/940536446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ff360 0x7fd4580ff740 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd448009880 tx=0x7fd44802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:42:25.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.993+0000 7fd45f357640 1 -- 172.21.15.82:0/940536446 shutdown_connections 2024-06-06T07:42:25.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.993+0000 7fd45f357640 1 --2- 172.21.15.82:0/940536446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4580ffd10 0x7fd458103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:25.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.993+0000 7fd45f357640 1 --2- 172.21.15.82:0/940536446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ff360 0x7fd4580ff740 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:25.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.993+0000 7fd45f357640 1 -- 172.21.15.82:0/940536446 >> 172.21.15.82:0/940536446 conn(0x7fd458075e10 msgr2=0x7fd4580fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:25.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.993+0000 7fd45f357640 1 -- 172.21.15.82:0/940536446 shutdown_connections 2024-06-06T07:42:25.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45f357640 1 -- 172.21.15.82:0/940536446 wait complete. 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45f357640 1 Processor -- start 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45f357640 1 -- start start 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45f357640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4580ff360 0x7fd458196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45f357640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ffd10 0x7fd458196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45f357640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd458196ef0 con 0x7fd4580ffd10 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45f357640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45819ac10 con 0x7fd4580ff360 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45c8f2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ffd10 0x7fd458196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45c8f2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ffd10 0x7fd458196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42836/0 (socket says 172.21.15.82:42836) 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.994+0000 7fd45c8f2640 1 -- 172.21.15.82:0/2204959367 learned_addr learned my addr 172.21.15.82:0/2204959367 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd45d0f3640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4580ff360 0x7fd458196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:25.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd45c8f2640 1 -- 172.21.15.82:0/2204959367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4580ff360 msgr2=0x7fd458196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd45c8f2640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4580ff360 0x7fd458196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd45c8f2640 1 -- 172.21.15.82:0/2204959367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd448009560 con 0x7fd4580ffd10 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd45c8f2640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ffd10 0x7fd458196860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd45400b3d0 tx=0x7fd45400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd44e7fc640 1 -- 172.21.15.82:0/2204959367 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd454011800 con 0x7fd4580ffd10 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd44e7fc640 1 -- 172.21.15.82:0/2204959367 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd454011e20 con 0x7fd4580ffd10 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45819aef0 con 0x7fd4580ffd10 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd44e7fc640 1 -- 172.21.15.82:0/2204959367 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd45400f2e0 con 0x7fd4580ffd10 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.995+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45819b440 con 0x7fd4580ffd10 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.996+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd458100840 con 0x7fd4580ffd10 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.996+0000 7fd44e7fc640 1 -- 172.21.15.82:0/2204959367 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fd454010380 con 0x7fd4580ffd10 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.997+0000 7fd44e7fc640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd44406aef0 0x7fd44406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.997+0000 7fd45d0f3640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd44406aef0 0x7fd44406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.997+0000 7fd44e7fc640 1 -- 172.21.15.82:0/2204959367 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd4540897e0 con 0x7fd4580ffd10 2024-06-06T07:42:25.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:24.997+0000 7fd45d0f3640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd44406aef0 0x7fd44406d3b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd448000c00 tx=0x7fd448004120 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:25.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.001+0000 7fd44e7fc640 1 -- 172.21.15.82:0/2204959367 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd454054e40 con 0x7fd4580ffd10 2024-06-06T07:42:25.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:24 smithi184 ceph-mon[37288]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:25.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.211+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd45804e210 con 0x7fd4580ffd10 2024-06-06T07:42:25.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.211+0000 7fd44e7fc640 1 -- 172.21.15.82:0/2204959367 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd454058460 con 0x7fd4580ffd10 2024-06-06T07:42:25.211 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:25.212 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:25.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd44406aef0 msgr2=0x7fd44406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:25.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd44406aef0 0x7fd44406d3b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd448000c00 tx=0x7fd448004120 comp rx=0 tx=0).stop 2024-06-06T07:42:25.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ffd10 msgr2=0x7fd458196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:25.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ffd10 0x7fd458196860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd45400b3d0 tx=0x7fd45400b890 comp rx=0 tx=0).stop 2024-06-06T07:42:25.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 shutdown_connections 2024-06-06T07:42:25.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd44406aef0 0x7fd44406d3b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:25.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd4580ffd10 0x7fd458196860 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:25.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 --2- 172.21.15.82:0/2204959367 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4580ff360 0x7fd458196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:25.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 >> 172.21.15.82:0/2204959367 conn(0x7fd458075e10 msgr2=0x7fd4580fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:25.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 shutdown_connections 2024-06-06T07:42:25.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:25.213+0000 7fd45f357640 1 -- 172.21.15.82:0/2204959367 wait complete. 2024-06-06T07:42:26.019 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:25 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2204959367' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:26.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:25 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2204959367' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:27.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:26 smithi082 ceph-mon[29354]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:27.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:26 smithi184 ceph-mon[37288]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:28.915 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:29.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:28 smithi082 ceph-mon[29354]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:29.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:28 smithi184 ceph-mon[37288]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:29.133 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:30.802 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:30 smithi082 ceph-mon[29354]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:30.885 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.882+0000 7f048d40f640 1 -- 172.21.15.82:0/1887928873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 msgr2=0x7f04880ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.882+0000 7f048d40f640 1 --2- 172.21.15.82:0/1887928873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 0x7f04880ff800 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0478009880 tx=0x7f047802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.882+0000 7f048d40f640 1 -- 172.21.15.82:0/1887928873 shutdown_connections 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.882+0000 7f048d40f640 1 --2- 172.21.15.82:0/1887928873 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04880ffdd0 0x7f0488103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.882+0000 7f048d40f640 1 --2- 172.21.15.82:0/1887928873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 0x7f04880ff800 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.882+0000 7f048d40f640 1 -- 172.21.15.82:0/1887928873 >> 172.21.15.82:0/1887928873 conn(0x7f04880facf0 msgr2=0x7f04880fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.882+0000 7f048d40f640 1 -- 172.21.15.82:0/1887928873 shutdown_connections 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.882+0000 7f048d40f640 1 -- 172.21.15.82:0/1887928873 wait complete. 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.882+0000 7f048d40f640 1 Processor -- start 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f048d40f640 1 -- start start 2024-06-06T07:42:30.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f048d40f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 0x7f04881971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f048d40f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04880ffdd0 0x7f0488197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f048d40f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488197c50 con 0x7f04880ff420 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f048d40f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488197dc0 con 0x7f04880ffdd0 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f0486ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 0x7f04881971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f0486ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 0x7f04881971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42858/0 (socket says 172.21.15.82:42858) 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f0486ffd640 1 -- 172.21.15.82:0/1687914543 learned_addr learned my addr 172.21.15.82:0/1687914543 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f0486ffd640 1 -- 172.21.15.82:0/1687914543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04880ffdd0 msgr2=0x7f0488197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f0486ffd640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04880ffdd0 0x7f0488197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f0486ffd640 1 -- 172.21.15.82:0/1687914543 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0478009560 con 0x7f04880ff420 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.883+0000 7f0486ffd640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 0x7f04881971d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f047802a9c0 tx=0x7f04780055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.884+0000 7f0484ff9640 1 -- 172.21.15.82:0/1687914543 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0478004aa0 con 0x7f04880ff420 2024-06-06T07:42:30.887 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.884+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0488198040 con 0x7f04880ff420 2024-06-06T07:42:30.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.884+0000 7f0484ff9640 1 -- 172.21.15.82:0/1687914543 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0478004c00 con 0x7f04880ff420 2024-06-06T07:42:30.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.884+0000 7f0484ff9640 1 -- 172.21.15.82:0/1687914543 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0478003ab0 con 0x7f04880ff420 2024-06-06T07:42:30.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.884+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04881a0520 con 0x7f04880ff420 2024-06-06T07:42:30.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.885+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0488100900 con 0x7f04880ff420 2024-06-06T07:42:30.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.885+0000 7f0484ff9640 1 -- 172.21.15.82:0/1687914543 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0478003ce0 con 0x7f04880ff420 2024-06-06T07:42:30.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.886+0000 7f0484ff9640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f047406aef0 0x7f047406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:30.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.886+0000 7f0484ff9640 1 -- 172.21.15.82:0/1687914543 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f04780aac50 con 0x7f04880ff420 2024-06-06T07:42:30.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.886+0000 7f047e7fc640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f047406aef0 0x7f047406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:30.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.886+0000 7f047e7fc640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f047406aef0 0x7f047406d3b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f04880fcec0 tx=0x7f0470006c90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:30.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:30.890+0000 7f0484ff9640 1 -- 172.21.15.82:0/1687914543 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f04780761a0 con 0x7f04880ff420 2024-06-06T07:42:31.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:30 smithi184 ceph-mon[37288]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:31.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.099+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0488198040 con 0x7f04880ff420 2024-06-06T07:42:31.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.100+0000 7f0484ff9640 1 -- 172.21.15.82:0/1687914543 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0488198040 con 0x7f04880ff420 2024-06-06T07:42:31.100 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:31.100 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f047406aef0 msgr2=0x7f047406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f047406aef0 0x7f047406d3b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f04880fcec0 tx=0x7f0470006c90 comp rx=0 tx=0).stop 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 msgr2=0x7f04881971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 0x7f04881971d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f047802a9c0 tx=0x7f04780055a0 comp rx=0 tx=0).stop 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 shutdown_connections 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f047406aef0 0x7f047406d3b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04880ffdd0 0x7f0488197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 --2- 172.21.15.82:0/1687914543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04880ff420 0x7f04881971d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 >> 172.21.15.82:0/1687914543 conn(0x7f04880facf0 msgr2=0x7f04880fc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:31.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 shutdown_connections 2024-06-06T07:42:31.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:31.102+0000 7f048d40f640 1 -- 172.21.15.82:0/1687914543 wait complete. 2024-06-06T07:42:32.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:31 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1687914543' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:32.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:31 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1687914543' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:33.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:32 smithi082 ceph-mon[29354]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:33.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:32 smithi184 ceph-mon[37288]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:34.722 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:34.937 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:35.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:34 smithi082 ceph-mon[29354]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:35.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:34 smithi184 ceph-mon[37288]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.526+0000 7f86cab7a640 1 -- 172.21.15.82:0/3570765578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ff3a0 msgr2=0x7f86c40ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.526+0000 7f86cab7a640 1 --2- 172.21.15.82:0/3570765578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ff3a0 0x7f86c40ff780 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f86b8009880 tx=0x7f86b802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.526+0000 7f86cab7a640 1 -- 172.21.15.82:0/3570765578 shutdown_connections 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.526+0000 7f86cab7a640 1 --2- 172.21.15.82:0/3570765578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86c40ffd50 0x7f86c4103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.526+0000 7f86cab7a640 1 --2- 172.21.15.82:0/3570765578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ff3a0 0x7f86c40ff780 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.526+0000 7f86cab7a640 1 -- 172.21.15.82:0/3570765578 >> 172.21.15.82:0/3570765578 conn(0x7f86c40fac50 msgr2=0x7f86c40fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.526+0000 7f86cab7a640 1 -- 172.21.15.82:0/3570765578 shutdown_connections 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.526+0000 7f86cab7a640 1 -- 172.21.15.82:0/3570765578 wait complete. 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.527+0000 7f86cab7a640 1 Processor -- start 2024-06-06T07:42:36.531 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.527+0000 7f86cab7a640 1 -- start start 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.527+0000 7f86cab7a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86c40ff3a0 0x7f86c4195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.527+0000 7f86cab7a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ffd50 0x7f86c4195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.527+0000 7f86cab7a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c4195ef0 con 0x7f86c40ffd50 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.527+0000 7f86cab7a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c4199c10 con 0x7f86c40ff3a0 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.527+0000 7f86c3fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ffd50 0x7f86c4195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.527+0000 7f86c3fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ffd50 0x7f86c4195860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57268/0 (socket says 172.21.15.82:57268) 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.527+0000 7f86c3fff640 1 -- 172.21.15.82:0/3727957548 learned_addr learned my addr 172.21.15.82:0/3727957548 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86c3fff640 1 -- 172.21.15.82:0/3727957548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86c40ff3a0 msgr2=0x7f86c4195320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86c8916640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86c40ff3a0 0x7f86c4195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86c3fff640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86c40ff3a0 0x7f86c4195320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86c3fff640 1 -- 172.21.15.82:0/3727957548 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b8009560 con 0x7f86c40ffd50 2024-06-06T07:42:36.532 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86c3fff640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ffd50 0x7f86c4195860 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f86b000b3d0 tx=0x7f86b000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:36.533 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86c1ffb640 1 -- 172.21.15.82:0/3727957548 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b0011800 con 0x7f86c40ffd50 2024-06-06T07:42:36.533 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c4199ef0 con 0x7f86c40ffd50 2024-06-06T07:42:36.533 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86c1ffb640 1 -- 172.21.15.82:0/3727957548 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86b0011e20 con 0x7f86c40ffd50 2024-06-06T07:42:36.533 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86c1ffb640 1 -- 172.21.15.82:0/3727957548 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b000f2e0 con 0x7f86c40ffd50 2024-06-06T07:42:36.533 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.528+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c419a440 con 0x7f86c40ffd50 2024-06-06T07:42:36.533 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.529+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c4100880 con 0x7f86c40ffd50 2024-06-06T07:42:36.533 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.529+0000 7f86c1ffb640 1 -- 172.21.15.82:0/3727957548 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f86b0010380 con 0x7f86c40ffd50 2024-06-06T07:42:36.534 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.530+0000 7f86c1ffb640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f86b406aef0 0x7f86b406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:36.534 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.530+0000 7f86c1ffb640 1 -- 172.21.15.82:0/3727957548 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f86b0089960 con 0x7f86c40ffd50 2024-06-06T07:42:36.534 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.530+0000 7f86c8916640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f86b406aef0 0x7f86b406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:36.534 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.530+0000 7f86c8916640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f86b406aef0 0x7f86b406d3b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f86b8000c00 tx=0x7f86b8004120 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:36.535 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.534+0000 7f86c1ffb640 1 -- 172.21.15.82:0/3727957548 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f86b0054f40 con 0x7f86c40ffd50 2024-06-06T07:42:36.749 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.748+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f86c404e210 con 0x7f86c40ffd50 2024-06-06T07:42:36.749 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.749+0000 7f86c1ffb640 1 -- 172.21.15.82:0/3727957548 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f86b0058560 con 0x7f86c40ffd50 2024-06-06T07:42:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:36.750 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:36.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f86b406aef0 msgr2=0x7f86b406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:36.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f86b406aef0 0x7f86b406d3b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f86b8000c00 tx=0x7f86b8004120 comp rx=0 tx=0).stop 2024-06-06T07:42:36.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ffd50 msgr2=0x7f86c4195860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:36.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ffd50 0x7f86c4195860 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f86b000b3d0 tx=0x7f86b000b890 comp rx=0 tx=0).stop 2024-06-06T07:42:36.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 shutdown_connections 2024-06-06T07:42:36.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f86b406aef0 0x7f86b406d3b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:36.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86c40ffd50 0x7f86c4195860 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:36.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 --2- 172.21.15.82:0/3727957548 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f86c40ff3a0 0x7f86c4195320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:36.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 >> 172.21.15.82:0/3727957548 conn(0x7f86c40fac50 msgr2=0x7f86c40fbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:36.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 shutdown_connections 2024-06-06T07:42:36.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:36.751+0000 7f86cab7a640 1 -- 172.21.15.82:0/3727957548 wait complete. 2024-06-06T07:42:37.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:36 smithi082 ceph-mon[29354]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:37.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:36 smithi184 ceph-mon[37288]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:38.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:37 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3727957548' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:38.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:42:38.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:37 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3727957548' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:38.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:42:39.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:38 smithi082 ceph-mon[29354]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:39.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:38 smithi184 ceph-mon[37288]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:40.409 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:40.627 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:41.003 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:40 smithi082 ceph-mon[29354]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:41.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:40 smithi184 ceph-mon[37288]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:42.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.230+0000 7fa79c6f0640 1 -- 172.21.15.82:0/2121207140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 msgr2=0x7fa794100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:42.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.230+0000 7fa79c6f0640 1 --2- 172.21.15.82:0/2121207140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 0x7fa794100850 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa790009880 tx=0x7fa79002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:42:42.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.230+0000 7fa79c6f0640 1 -- 172.21.15.82:0/2121207140 shutdown_connections 2024-06-06T07:42:42.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.230+0000 7fa79c6f0640 1 --2- 172.21.15.82:0/2121207140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa794100e20 0x7fa794104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:42.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.230+0000 7fa79c6f0640 1 --2- 172.21.15.82:0/2121207140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 0x7fa794100850 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:42.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.230+0000 7fa79c6f0640 1 -- 172.21.15.82:0/2121207140 >> 172.21.15.82:0/2121207140 conn(0x7fa7940fbd20 msgr2=0x7fa7940fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:42.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.230+0000 7fa79c6f0640 1 -- 172.21.15.82:0/2121207140 shutdown_connections 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.231+0000 7fa79c6f0640 1 -- 172.21.15.82:0/2121207140 wait complete. 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.231+0000 7fa79c6f0640 1 Processor -- start 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.231+0000 7fa79c6f0640 1 -- start start 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.231+0000 7fa79c6f0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 0x7fa794196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.231+0000 7fa79c6f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa794100e20 0x7fa794196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.231+0000 7fa79c6f0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794196f00 con 0x7fa794100470 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.231+0000 7fa79c6f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79419ac20 con 0x7fa794100e20 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.231+0000 7fa79a48c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 0x7fa794196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.231+0000 7fa79a48c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 0x7fa794196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57288/0 (socket says 172.21.15.82:57288) 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa79a48c640 1 -- 172.21.15.82:0/4141364280 learned_addr learned my addr 172.21.15.82:0/4141364280 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa799c8b640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa794100e20 0x7fa794196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa79a48c640 1 -- 172.21.15.82:0/4141364280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa794100e20 msgr2=0x7fa794196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa79a48c640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa794100e20 0x7fa794196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa79a48c640 1 -- 172.21.15.82:0/4141364280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790009560 con 0x7fa794100470 2024-06-06T07:42:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa799c8b640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa794100e20 0x7fa794196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:42:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa79a48c640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 0x7fa794196330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa79002a9c0 tx=0x7fa7900055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa78b7fe640 1 -- 172.21.15.82:0/4141364280 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa790004aa0 con 0x7fa794100470 2024-06-06T07:42:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79419aea0 con 0x7fa794100470 2024-06-06T07:42:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa78b7fe640 1 -- 172.21.15.82:0/4141364280 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa790004c00 con 0x7fa794100470 2024-06-06T07:42:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa78b7fe640 1 -- 172.21.15.82:0/4141364280 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa790003ab0 con 0x7fa794100470 2024-06-06T07:42:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.232+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79419b390 con 0x7fa794100470 2024-06-06T07:42:42.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.233+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa794101950 con 0x7fa794100470 2024-06-06T07:42:42.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.234+0000 7fa78b7fe640 1 -- 172.21.15.82:0/4141364280 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fa790031d20 con 0x7fa794100470 2024-06-06T07:42:42.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.234+0000 7fa78b7fe640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa78006aef0 0x7fa78006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:42.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.234+0000 7fa78b7fe640 1 -- 172.21.15.82:0/4141364280 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa7900aad10 con 0x7fa794100470 2024-06-06T07:42:42.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.234+0000 7fa799c8b640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa78006aef0 0x7fa78006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:42.236 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.235+0000 7fa799c8b640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa78006aef0 0x7fa78006d3b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa784006fd0 tx=0x7fa784008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:42.238 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.238+0000 7fa78b7fe640 1 -- 172.21.15.82:0/4141364280 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa7900762f0 con 0x7fa794100470 2024-06-06T07:42:42.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.445+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa79404e210 con 0x7fa794100470 2024-06-06T07:42:42.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.445+0000 7fa78b7fe640 1 -- 172.21.15.82:0/4141364280 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa790079910 con 0x7fa794100470 2024-06-06T07:42:42.445 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:42.446 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:42.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.447+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa78006aef0 msgr2=0x7fa78006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:42.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa78006aef0 0x7fa78006d3b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa784006fd0 tx=0x7fa784008040 comp rx=0 tx=0).stop 2024-06-06T07:42:42.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 msgr2=0x7fa794196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:42.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 0x7fa794196330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa79002a9c0 tx=0x7fa7900055a0 comp rx=0 tx=0).stop 2024-06-06T07:42:42.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 shutdown_connections 2024-06-06T07:42:42.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fa78006aef0 0x7fa78006d3b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:42.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa794100e20 0x7fa794196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:42.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 --2- 172.21.15.82:0/4141364280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa794100470 0x7fa794196330 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:42.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 >> 172.21.15.82:0/4141364280 conn(0x7fa7940fbd20 msgr2=0x7fa7940fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:42.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 shutdown_connections 2024-06-06T07:42:42.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:42.448+0000 7fa79c6f0640 1 -- 172.21.15.82:0/4141364280 wait complete. 2024-06-06T07:42:43.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:42 smithi082 ceph-mon[29354]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:43.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:42 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/4141364280' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:43.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:42 smithi184 ceph-mon[37288]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:43.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:42 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/4141364280' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:45.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:44 smithi082 ceph-mon[29354]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:45.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:44 smithi184 ceph-mon[37288]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:46.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:46.448 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:46.866 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:46 smithi082 ceph-mon[29354]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:47.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:46 smithi184 ceph-mon[37288]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:47.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.977+0000 7fbc0769d640 1 -- 172.21.15.82:0/2685977372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 msgr2=0x7fbc00110d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:47.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.977+0000 7fbc0769d640 1 --2- 172.21.15.82:0/2685977372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 0x7fbc00110d20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009880 tx=0x7fbbfc02a530 comp rx=0 tx=0).stop 2024-06-06T07:42:47.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.977+0000 7fbc0769d640 1 -- 172.21.15.82:0/2685977372 shutdown_connections 2024-06-06T07:42:47.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.977+0000 7fbc0769d640 1 --2- 172.21.15.82:0/2685977372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 0x7fbc00110d20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:47.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.977+0000 7fbc0769d640 1 --2- 172.21.15.82:0/2685977372 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc00072850 0x7fbc00072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.977+0000 7fbc0769d640 1 -- 172.21.15.82:0/2685977372 >> 172.21.15.82:0/2685977372 conn(0x7fbc000fbda0 msgr2=0x7fbc000fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.977+0000 7fbc0769d640 1 -- 172.21.15.82:0/2685977372 shutdown_connections 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.977+0000 7fbc0769d640 1 -- 172.21.15.82:0/2685977372 wait complete. 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc0769d640 1 Processor -- start 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc0769d640 1 -- start start 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc0769d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc00072850 0x7fbc001a2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc0769d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 0x7fbc001a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc0769d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0019c710 con 0x7fbc00073170 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc0769d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0019c850 con 0x7fbc00072850 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc04c38640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 0x7fbc001a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc04c38640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 0x7fbc001a2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33856/0 (socket says 172.21.15.82:33856) 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc04c38640 1 -- 172.21.15.82:0/2292262660 learned_addr learned my addr 172.21.15.82:0/2292262660 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:47.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc05439640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc00072850 0x7fbc001a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc04c38640 1 -- 172.21.15.82:0/2292262660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc00072850 msgr2=0x7fbc001a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc04c38640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc00072850 0x7fbc001a2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.978+0000 7fbc04c38640 1 -- 172.21.15.82:0/2292262660 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc009560 con 0x7fbc00073170 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.979+0000 7fbc04c38640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 0x7fbc001a2ba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc02aa50 tx=0x7fbbfc0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.979+0000 7fbbf67fc640 1 -- 172.21.15.82:0/2292262660 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbfc004aa0 con 0x7fbc00073170 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.979+0000 7fbbf67fc640 1 -- 172.21.15.82:0/2292262660 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbfc004c00 con 0x7fbc00073170 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.979+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0019cb30 con 0x7fbc00073170 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.979+0000 7fbbf67fc640 1 -- 172.21.15.82:0/2292262660 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbfc003ab0 con 0x7fbc00073170 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.979+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0019d080 con 0x7fbc00073170 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.980+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc00073f90 con 0x7fbc00073170 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.980+0000 7fbbf67fc640 1 -- 172.21.15.82:0/2292262660 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fbbfc02ad50 con 0x7fbc00073170 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.980+0000 7fbbf67fc640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fbbec06aef0 0x7fbbec06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:47.983 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.981+0000 7fbbf67fc640 1 -- 172.21.15.82:0/2292262660 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbbfc0aa960 con 0x7fbc00073170 2024-06-06T07:42:47.984 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.981+0000 7fbc05439640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fbbec06aef0 0x7fbbec06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:47.984 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.981+0000 7fbc05439640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fbbec06aef0 0x7fbbec06d3b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009890 tx=0x7fbbf0008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:47.985 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:47.985+0000 7fbbf67fc640 1 -- 172.21.15.82:0/2292262660 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fbbfc075fc0 con 0x7fbc00073170 2024-06-06T07:42:48.193 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.193+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbc00113af0 con 0x7fbc00073170 2024-06-06T07:42:48.193 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.193+0000 7fbbf67fc640 1 -- 172.21.15.82:0/2292262660 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbbfc0795e0 con 0x7fbc00073170 2024-06-06T07:42:48.193 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:48.194 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:48.195 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fbbec06aef0 msgr2=0x7fbbec06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fbbec06aef0 0x7fbbec06d3b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009890 tx=0x7fbbf0008040 comp rx=0 tx=0).stop 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 msgr2=0x7fbc001a2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 0x7fbc001a2ba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc02aa50 tx=0x7fbbfc0055a0 comp rx=0 tx=0).stop 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 shutdown_connections 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fbbec06aef0 0x7fbbec06d3b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc00073170 0x7fbc001a2ba0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 --2- 172.21.15.82:0/2292262660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc00072850 0x7fbc001a2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 >> 172.21.15.82:0/2292262660 conn(0x7fbc000fbda0 msgr2=0x7fbc000fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 shutdown_connections 2024-06-06T07:42:48.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:48.195+0000 7fbc0769d640 1 -- 172.21.15.82:0/2292262660 wait complete. 2024-06-06T07:42:49.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:48 smithi082 ceph-mon[29354]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:49.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:48 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2292262660' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:49.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:48 smithi184 ceph-mon[37288]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:49.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:48 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2292262660' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:51.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:50 smithi082 ceph-mon[29354]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:51.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:50 smithi184 ceph-mon[37288]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:51.849 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:52.066 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:52.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:52 smithi082 ceph-mon[29354]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:52.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:42:53.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:52 smithi184 ceph-mon[37288]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.638+0000 7f11f0604640 1 -- 172.21.15.82:0/3329783236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100470 msgr2=0x7f11e8100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.638+0000 7f11f0604640 1 --2- 172.21.15.82:0/3329783236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100470 0x7f11e8100850 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009880 tx=0x7f11e402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.638+0000 7f11f0604640 1 -- 172.21.15.82:0/3329783236 shutdown_connections 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.638+0000 7f11f0604640 1 --2- 172.21.15.82:0/3329783236 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11e8100e20 0x7f11e8104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.638+0000 7f11f0604640 1 --2- 172.21.15.82:0/3329783236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100470 0x7f11e8100850 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.638+0000 7f11f0604640 1 -- 172.21.15.82:0/3329783236 >> 172.21.15.82:0/3329783236 conn(0x7f11e80fbd20 msgr2=0x7f11e80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.638+0000 7f11f0604640 1 -- 172.21.15.82:0/3329783236 shutdown_connections 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.638+0000 7f11f0604640 1 -- 172.21.15.82:0/3329783236 wait complete. 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11f0604640 1 Processor -- start 2024-06-06T07:42:53.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11f0604640 1 -- start start 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11f0604640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11e8100470 0x7f11e8196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11f0604640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100e20 0x7f11e81967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11f0604640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e8196e40 con 0x7f11e8100e20 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11f0604640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e819ab10 con 0x7f11e8100470 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11edb9f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100e20 0x7f11e81967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11edb9f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100e20 0x7f11e81967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:33864/0 (socket says 172.21.15.82:33864) 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11edb9f640 1 -- 172.21.15.82:0/1836053582 learned_addr learned my addr 172.21.15.82:0/1836053582 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11ee3a0640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11e8100470 0x7f11e8196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11edb9f640 1 -- 172.21.15.82:0/1836053582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11e8100470 msgr2=0x7f11e8196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11edb9f640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11e8100470 0x7f11e8196270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.639+0000 7f11edb9f640 1 -- 172.21.15.82:0/1836053582 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e4009560 con 0x7f11e8100e20 2024-06-06T07:42:53.661 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.640+0000 7f11edb9f640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100e20 0x7f11e81967b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c8f0 tx=0x7f11d800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.640+0000 7f11df7fe640 1 -- 172.21.15.82:0/1836053582 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11d8007b80 con 0x7f11e8100e20 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.640+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e819adf0 con 0x7f11e8100e20 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.640+0000 7f11df7fe640 1 -- 172.21.15.82:0/1836053582 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11d80054e0 con 0x7f11e8100e20 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.640+0000 7f11df7fe640 1 -- 172.21.15.82:0/1836053582 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11d800f450 con 0x7f11e8100e20 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.640+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e819b340 con 0x7f11e8100e20 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.641+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e8101950 con 0x7f11e8100e20 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.641+0000 7f11df7fe640 1 -- 172.21.15.82:0/1836053582 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f11d8004f80 con 0x7f11e8100e20 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.642+0000 7f11df7fe640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f11d406aef0 0x7f11d406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.642+0000 7f11ee3a0640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f11d406aef0 0x7f11d406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.642+0000 7f11df7fe640 1 -- 172.21.15.82:0/1836053582 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f11d8089a20 con 0x7f11e8100e20 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.642+0000 7f11ee3a0640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f11d406aef0 0x7f11d406d3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f11e4000c00 tx=0x7f11e4005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:53.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.646+0000 7f11df7fe640 1 -- 172.21.15.82:0/1836053582 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f11d8055000 con 0x7f11e8100e20 2024-06-06T07:42:53.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.856+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f11e804e210 con 0x7f11e8100e20 2024-06-06T07:42:53.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.856+0000 7f11df7fe640 1 -- 172.21.15.82:0/1836053582 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f11d801d0d0 con 0x7f11e8100e20 2024-06-06T07:42:53.858 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:53.858 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:53.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.858+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f11d406aef0 msgr2=0x7f11d406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.858+0000 7f11f0604640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f11d406aef0 0x7f11d406d3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f11e4000c00 tx=0x7f11e4005a20 comp rx=0 tx=0).stop 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.858+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100e20 msgr2=0x7f11e81967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.858+0000 7f11f0604640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100e20 0x7f11e81967b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c8f0 tx=0x7f11d800cdb0 comp rx=0 tx=0).stop 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.859+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 shutdown_connections 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.859+0000 7f11f0604640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f11d406aef0 0x7f11d406d3b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.859+0000 7f11f0604640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8100e20 0x7f11e81967b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.859+0000 7f11f0604640 1 --2- 172.21.15.82:0/1836053582 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11e8100470 0x7f11e8196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.859+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 >> 172.21.15.82:0/1836053582 conn(0x7f11e80fbd20 msgr2=0x7f11e80fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.859+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 shutdown_connections 2024-06-06T07:42:53.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:53.859+0000 7f11f0604640 1 -- 172.21.15.82:0/1836053582 wait complete. 2024-06-06T07:42:55.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:54 smithi082 ceph-mon[29354]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:55.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:54 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1836053582' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:55.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:54 smithi184 ceph-mon[37288]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:55.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:54 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1836053582' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:42:57.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:56 smithi082 ceph-mon[29354]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:57.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:56 smithi184 ceph-mon[37288]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:57.502 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:42:57.721 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:42:59.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:58 smithi082 ceph-mon[29354]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:59.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:58 smithi184 ceph-mon[37288]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:42:59.342 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.339+0000 7f923c82a640 1 -- 172.21.15.82:0/1768740982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ff360 msgr2=0x7f92380ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:59.342 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.339+0000 7f923c82a640 1 --2- 172.21.15.82:0/1768740982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ff360 0x7f92380ff740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9228009880 tx=0x7f922802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.339+0000 7f923c82a640 1 -- 172.21.15.82:0/1768740982 shutdown_connections 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.339+0000 7f923c82a640 1 --2- 172.21.15.82:0/1768740982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92380ffd10 0x7f9238103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.339+0000 7f923c82a640 1 --2- 172.21.15.82:0/1768740982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ff360 0x7f92380ff740 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.339+0000 7f923c82a640 1 -- 172.21.15.82:0/1768740982 >> 172.21.15.82:0/1768740982 conn(0x7f9238075e10 msgr2=0x7f92380fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.339+0000 7f923c82a640 1 -- 172.21.15.82:0/1768740982 shutdown_connections 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.339+0000 7f923c82a640 1 -- 172.21.15.82:0/1768740982 wait complete. 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.340+0000 7f923c82a640 1 Processor -- start 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.340+0000 7f923c82a640 1 -- start start 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.340+0000 7f923c82a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92380ff360 0x7f92381970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.340+0000 7f923c82a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ffd10 0x7f9238197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.340+0000 7f923c82a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9238197b50 con 0x7f92380ffd10 2024-06-06T07:42:59.343 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.340+0000 7f923c82a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9238197cc0 con 0x7f92380ff360 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.340+0000 7f9235d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ffd10 0x7f9238197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f9235d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ffd10 0x7f9238197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:35624/0 (socket says 172.21.15.82:35624) 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f9235d9b640 1 -- 172.21.15.82:0/2206789089 learned_addr learned my addr 172.21.15.82:0/2206789089 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f923659c640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92380ff360 0x7f92381970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f9235d9b640 1 -- 172.21.15.82:0/2206789089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92380ff360 msgr2=0x7f92381970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f9235d9b640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92380ff360 0x7f92381970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f9235d9b640 1 -- 172.21.15.82:0/2206789089 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9228009560 con 0x7f92380ffd10 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f9235d9b640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ffd10 0x7f9238197610 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f922c00c8f0 tx=0x7f922c00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f92277fe640 1 -- 172.21.15.82:0/2206789089 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922c007b80 con 0x7f92380ffd10 2024-06-06T07:42:59.344 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f92277fe640 1 -- 172.21.15.82:0/2206789089 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f922c007ce0 con 0x7f92380ffd10 2024-06-06T07:42:59.345 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f92277fe640 1 -- 172.21.15.82:0/2206789089 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922c0187b0 con 0x7f92380ffd10 2024-06-06T07:42:59.345 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92381a0140 con 0x7f92380ffd10 2024-06-06T07:42:59.345 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.341+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92381a0500 con 0x7f92380ffd10 2024-06-06T07:42:59.346 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.342+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9238100840 con 0x7f92380ffd10 2024-06-06T07:42:59.346 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.343+0000 7f92277fe640 1 -- 172.21.15.82:0/2206789089 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f922c00f450 con 0x7f92380ffd10 2024-06-06T07:42:59.346 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.343+0000 7f92277fe640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f922006aef0 0x7f922006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:42:59.346 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.343+0000 7f92277fe640 1 -- 172.21.15.82:0/2206789089 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f922c089d50 con 0x7f92380ffd10 2024-06-06T07:42:59.346 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.343+0000 7f923659c640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f922006aef0 0x7f922006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:42:59.346 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.344+0000 7f923659c640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f922006aef0 0x7f922006d3b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9228000c00 tx=0x7f9228004120 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:42:59.348 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.347+0000 7f92277fe640 1 -- 172.21.15.82:0/2206789089 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f922c055330 con 0x7f92380ffd10 2024-06-06T07:42:59.557 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.556+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f923804e210 con 0x7f92380ffd10 2024-06-06T07:42:59.557 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.556+0000 7f92277fe640 1 -- 172.21.15.82:0/2206789089 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f922c058950 con 0x7f92380ffd10 2024-06-06T07:42:59.557 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:42:59.557 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:42:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f922006aef0 msgr2=0x7f922006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f922006aef0 0x7f922006d3b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9228000c00 tx=0x7f9228004120 comp rx=0 tx=0).stop 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ffd10 msgr2=0x7f9238197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ffd10 0x7f9238197610 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f922c00c8f0 tx=0x7f922c00cdb0 comp rx=0 tx=0).stop 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 shutdown_connections 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f922006aef0 0x7f922006d3b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f92380ffd10 0x7f9238197610 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 --2- 172.21.15.82:0/2206789089 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92380ff360 0x7f92381970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 >> 172.21.15.82:0/2206789089 conn(0x7f9238075e10 msgr2=0x7f9238102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 shutdown_connections 2024-06-06T07:42:59.560 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:42:59.558+0000 7f923c82a640 1 -- 172.21.15.82:0/2206789089 wait complete. 2024-06-06T07:43:00.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:42:59 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2206789089' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:00.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:42:59 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2206789089' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:01.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:00 smithi082 ceph-mon[29354]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:01.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:00 smithi184 ceph-mon[37288]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:03.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:02 smithi082 ceph-mon[29354]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:03.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:02 smithi184 ceph-mon[37288]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:03.189 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:03.407 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:04.941 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 -- 172.21.15.82:0/391746861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b00fe9f0 msgr2=0x7f08b00fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:04.941 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 --2- 172.21.15.82:0/391746861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b00fe9f0 0x7f08b00fee30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0898009880 tx=0x7f089802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 -- 172.21.15.82:0/391746861 shutdown_connections 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 --2- 172.21.15.82:0/391746861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b00fe9f0 0x7f08b00fee30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 --2- 172.21.15.82:0/391746861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08b01062f0 0x7f08b01066d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 -- 172.21.15.82:0/391746861 >> 172.21.15.82:0/391746861 conn(0x7f08b0074430 msgr2=0x7f08b0074840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 -- 172.21.15.82:0/391746861 shutdown_connections 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 -- 172.21.15.82:0/391746861 wait complete. 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 Processor -- start 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 -- start start 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08b00fe9f0 0x7f08b0196240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b01062f0 0x7f08b0196780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b0196e10 con 0x7f08b01062f0 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.936+0000 7f08b63cf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b019ab30 con 0x7f08b00fe9f0 2024-06-06T07:43:04.942 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08a7fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b01062f0 0x7f08b0196780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08a7fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b01062f0 0x7f08b0196780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:35632/0 (socket says 172.21.15.82:35632) 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08a7fff640 1 -- 172.21.15.82:0/2940318679 learned_addr learned my addr 172.21.15.82:0/2940318679 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08affff640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08b00fe9f0 0x7f08b0196240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08a7fff640 1 -- 172.21.15.82:0/2940318679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08b00fe9f0 msgr2=0x7f08b0196240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08a7fff640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08b00fe9f0 0x7f08b0196240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08a7fff640 1 -- 172.21.15.82:0/2940318679 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0898009560 con 0x7f08b01062f0 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08a7fff640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b01062f0 0x7f08b0196780 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0898006010 tx=0x7f08980055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08adffb640 1 -- 172.21.15.82:0/2940318679 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0898004aa0 con 0x7f08b01062f0 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08adffb640 1 -- 172.21.15.82:0/2940318679 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0898004c00 con 0x7f08b01062f0 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b019adb0 con 0x7f08b01062f0 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08adffb640 1 -- 172.21.15.82:0/2940318679 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0898003a20 con 0x7f08b01062f0 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.937+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b019b2a0 con 0x7f08b01062f0 2024-06-06T07:43:04.943 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.938+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b0068410 con 0x7f08b01062f0 2024-06-06T07:43:04.944 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.939+0000 7f08adffb640 1 -- 172.21.15.82:0/2940318679 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0898031c90 con 0x7f08b01062f0 2024-06-06T07:43:04.944 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.939+0000 7f08adffb640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f089c06af00 0x7f089c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:04.944 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.939+0000 7f08adffb640 1 -- 172.21.15.82:0/2940318679 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f08980aad70 con 0x7f08b01062f0 2024-06-06T07:43:04.944 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.939+0000 7f08affff640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f089c06af00 0x7f089c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:04.944 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.940+0000 7f08affff640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f089c06af00 0x7f089c06d3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f08a00058c0 tx=0x7f08a000a2d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:04.944 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:04.944+0000 7f08adffb640 1 -- 172.21.15.82:0/2940318679 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0898076400 con 0x7f08b01062f0 2024-06-06T07:43:05.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:04 smithi082 ceph-mon[29354]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:05.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:04 smithi184 ceph-mon[37288]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:05.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.151+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f08b0108760 con 0x7f08b01062f0 2024-06-06T07:43:05.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.151+0000 7f08adffb640 1 -- 172.21.15.82:0/2940318679 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0898079a20 con 0x7f08b01062f0 2024-06-06T07:43:05.152 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:05.152 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.153+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f089c06af00 msgr2=0x7f089c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.153+0000 7f08b63cf640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f089c06af00 0x7f089c06d3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f08a00058c0 tx=0x7f08a000a2d0 comp rx=0 tx=0).stop 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.153+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b01062f0 msgr2=0x7f08b0196780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.153+0000 7f08b63cf640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b01062f0 0x7f08b0196780 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0898006010 tx=0x7f08980055a0 comp rx=0 tx=0).stop 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.154+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 shutdown_connections 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.154+0000 7f08b63cf640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f089c06af00 0x7f089c06d3c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.154+0000 7f08b63cf640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08b01062f0 0x7f08b0196780 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.154+0000 7f08b63cf640 1 --2- 172.21.15.82:0/2940318679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08b00fe9f0 0x7f08b0196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.154+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 >> 172.21.15.82:0/2940318679 conn(0x7f08b0074430 msgr2=0x7f08b00fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:05.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.154+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 shutdown_connections 2024-06-06T07:43:05.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:05.154+0000 7f08b63cf640 1 -- 172.21.15.82:0/2940318679 wait complete. 2024-06-06T07:43:05.920 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:05 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2940318679' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:05.920 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:05 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:43:05.920 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:05 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:43:05.921 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:05 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:43:05.921 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:05 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:43:06.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:05 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2940318679' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:06.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:05 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:43:06.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:05 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:43:06.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:05 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:43:06.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:05 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:43:06.844 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:06 smithi184 ceph-mon[37288]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:07.032 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:06 smithi082 ceph-mon[29354]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:08.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:43:08.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:43:08.941 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:09.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:08 smithi184 ceph-mon[37288]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:09.155 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:09.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:08 smithi082 ceph-mon[29354]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:10.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:09 smithi082 ceph-mon[29354]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:10.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:09 smithi184 ceph-mon[37288]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:10.754 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.748+0000 7f27780f0640 1 -- 172.21.15.82:0/4253337698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 msgr2=0x7f277010c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:10.754 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.748+0000 7f27780f0640 1 --2- 172.21.15.82:0/4253337698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 0x7f277010c600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2760009880 tx=0x7f276002a490 comp rx=0 tx=0).stop 2024-06-06T07:43:10.754 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.748+0000 7f27780f0640 1 -- 172.21.15.82:0/4253337698 shutdown_connections 2024-06-06T07:43:10.754 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.748+0000 7f27780f0640 1 --2- 172.21.15.82:0/4253337698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 0x7f277010c600 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:10.754 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.748+0000 7f27780f0640 1 --2- 172.21.15.82:0/4253337698 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27700fef90 0x7f27700ff370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:10.754 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.748+0000 7f27780f0640 1 -- 172.21.15.82:0/4253337698 >> 172.21.15.82:0/4253337698 conn(0x7f27700fac50 msgr2=0x7f27700fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:10.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.748+0000 7f27780f0640 1 -- 172.21.15.82:0/4253337698 shutdown_connections 2024-06-06T07:43:10.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.748+0000 7f27780f0640 1 -- 172.21.15.82:0/4253337698 wait complete. 2024-06-06T07:43:10.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.749+0000 7f27780f0640 1 Processor -- start 2024-06-06T07:43:10.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.749+0000 7f27780f0640 1 -- start start 2024-06-06T07:43:10.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.749+0000 7f27780f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27700fef90 0x7f2770198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:10.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.749+0000 7f27780f0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 0x7f27701989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:10.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.749+0000 7f27780f0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2770199040 con 0x7f27700ff940 2024-06-06T07:43:10.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.749+0000 7f27780f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277019cd60 con 0x7f27700fef90 2024-06-06T07:43:10.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.749+0000 7f277568b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 0x7f27701989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:10.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.749+0000 7f277568b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 0x7f27701989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:34872/0 (socket says 172.21.15.82:34872) 2024-06-06T07:43:10.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.749+0000 7f277568b640 1 -- 172.21.15.82:0/2757488617 learned_addr learned my addr 172.21.15.82:0/2757488617 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:10.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.750+0000 7f277568b640 1 -- 172.21.15.82:0/2757488617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27700fef90 msgr2=0x7f2770198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:10.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.750+0000 7f277568b640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27700fef90 0x7f2770198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:10.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.750+0000 7f277568b640 1 -- 172.21.15.82:0/2757488617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2760009560 con 0x7f27700ff940 2024-06-06T07:43:10.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.750+0000 7f277568b640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 0x7f27701989b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f276002a9b0 tx=0x7f27600055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:10.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.750+0000 7f2766ffd640 1 -- 172.21.15.82:0/2757488617 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2760004aa0 con 0x7f27700ff940 2024-06-06T07:43:10.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.750+0000 7f2766ffd640 1 -- 172.21.15.82:0/2757488617 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2760004c00 con 0x7f27700ff940 2024-06-06T07:43:10.757 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.750+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277019d040 con 0x7f27700ff940 2024-06-06T07:43:10.757 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.750+0000 7f2766ffd640 1 -- 172.21.15.82:0/2757488617 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2760003ab0 con 0x7f27700ff940 2024-06-06T07:43:10.757 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.750+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277019d590 con 0x7f27700ff940 2024-06-06T07:43:10.757 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.751+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2770069080 con 0x7f27700ff940 2024-06-06T07:43:10.757 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.751+0000 7f2766ffd640 1 -- 172.21.15.82:0/2757488617 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f2760003ce0 con 0x7f27700ff940 2024-06-06T07:43:10.757 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.752+0000 7f2766ffd640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f275c06aef0 0x7f275c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:10.757 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.752+0000 7f2775e8c640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f275c06aef0 0x7f275c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:10.757 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.752+0000 7f2766ffd640 1 -- 172.21.15.82:0/2757488617 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f27600aace0 con 0x7f27700ff940 2024-06-06T07:43:10.757 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.752+0000 7f2775e8c640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f275c06aef0 0x7f275c06d3b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f276c009710 tx=0x7f276c006c90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:10.758 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.756+0000 7f2766ffd640 1 -- 172.21.15.82:0/2757488617 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f27600762c0 con 0x7f27700ff940 2024-06-06T07:43:10.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.965+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f277004e210 con 0x7f27700ff940 2024-06-06T07:43:10.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.966+0000 7f2766ffd640 1 -- 172.21.15.82:0/2757488617 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f27600798e0 con 0x7f27700ff940 2024-06-06T07:43:10.966 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:10.966 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:10.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f275c06aef0 msgr2=0x7f275c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:10.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f275c06aef0 0x7f275c06d3b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f276c009710 tx=0x7f276c006c90 comp rx=0 tx=0).stop 2024-06-06T07:43:10.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 msgr2=0x7f27701989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:10.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 0x7f27701989b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f276002a9b0 tx=0x7f27600055a0 comp rx=0 tx=0).stop 2024-06-06T07:43:10.971 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 shutdown_connections 2024-06-06T07:43:10.971 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f275c06aef0 0x7f275c06d3b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:10.971 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27700ff940 0x7f27701989b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:10.971 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 --2- 172.21.15.82:0/2757488617 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27700fef90 0x7f2770198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:10.971 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 >> 172.21.15.82:0/2757488617 conn(0x7f27700fac50 msgr2=0x7f27700fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:10.971 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 shutdown_connections 2024-06-06T07:43:10.971 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:10.968+0000 7f27780f0640 1 -- 172.21.15.82:0/2757488617 wait complete. 2024-06-06T07:43:11.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:11 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2757488617' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:11.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:11 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2757488617' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:12.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:12 smithi184 ceph-mon[37288]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:12.517 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:12 smithi082 ceph-mon[29354]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:14.569 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:14.787 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:14.909 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:14 smithi082 ceph-mon[29354]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:15.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:14 smithi184 ceph-mon[37288]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:16.491 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.487+0000 7fe3de29f640 1 -- 172.21.15.82:0/2778557837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ff3a0 msgr2=0x7fe3d80ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.487+0000 7fe3de29f640 1 --2- 172.21.15.82:0/2778557837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ff3a0 0x7fe3d80ff780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009880 tx=0x7fe3c802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.487+0000 7fe3de29f640 1 -- 172.21.15.82:0/2778557837 shutdown_connections 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.487+0000 7fe3de29f640 1 --2- 172.21.15.82:0/2778557837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3d80ffd50 0x7fe3d8103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.487+0000 7fe3de29f640 1 --2- 172.21.15.82:0/2778557837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ff3a0 0x7fe3d80ff780 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.487+0000 7fe3de29f640 1 -- 172.21.15.82:0/2778557837 >> 172.21.15.82:0/2778557837 conn(0x7fe3d80fac50 msgr2=0x7fe3d80fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.487+0000 7fe3de29f640 1 -- 172.21.15.82:0/2778557837 shutdown_connections 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.487+0000 7fe3de29f640 1 -- 172.21.15.82:0/2778557837 wait complete. 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3de29f640 1 Processor -- start 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3de29f640 1 -- start start 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3de29f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3d80ff3a0 0x7fe3d8196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3de29f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ffd50 0x7fe3d81967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3de29f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d8196da0 con 0x7fe3d80ffd50 2024-06-06T07:43:16.492 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3de29f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d8196f10 con 0x7fe3d80ff3a0 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3d77fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ffd50 0x7fe3d81967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3d77fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ffd50 0x7fe3d81967a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:37254/0 (socket says 172.21.15.82:37254) 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3d77fe640 1 -- 172.21.15.82:0/3525090821 learned_addr learned my addr 172.21.15.82:0/3525090821 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3d77fe640 1 -- 172.21.15.82:0/3525090821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3d80ff3a0 msgr2=0x7fe3d8196260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3d77fe640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3d80ff3a0 0x7fe3d8196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3d77fe640 1 -- 172.21.15.82:0/3525090821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c8009560 con 0x7fe3d80ffd50 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.488+0000 7fe3d77fe640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ffd50 0x7fe3d81967a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c8f0 tx=0x7fe3cc00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.489+0000 7fe3d57fa640 1 -- 172.21.15.82:0/3525090821 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3cc004410 con 0x7fe3d80ffd50 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.489+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d819ac50 con 0x7fe3d80ffd50 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.489+0000 7fe3d57fa640 1 -- 172.21.15.82:0/3525090821 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3cc007c60 con 0x7fe3d80ffd50 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.489+0000 7fe3d57fa640 1 -- 172.21.15.82:0/3525090821 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3cc00f450 con 0x7fe3d80ffd50 2024-06-06T07:43:16.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.489+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d819b1a0 con 0x7fe3d80ffd50 2024-06-06T07:43:16.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.490+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d8100880 con 0x7fe3d80ffd50 2024-06-06T07:43:16.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.490+0000 7fe3d57fa640 1 -- 172.21.15.82:0/3525090821 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fe3cc0077a0 con 0x7fe3d80ffd50 2024-06-06T07:43:16.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.490+0000 7fe3d57fa640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe3c006aef0 0x7fe3c006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:16.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.491+0000 7fe3d57fa640 1 -- 172.21.15.82:0/3525090821 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe3cc089890 con 0x7fe3d80ffd50 2024-06-06T07:43:16.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.491+0000 7fe3d7fff640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe3c006aef0 0x7fe3c006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:16.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.491+0000 7fe3d7fff640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe3c006aef0 0x7fe3c006d3b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009880 tx=0x7fe3c8000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:16.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.495+0000 7fe3d57fa640 1 -- 172.21.15.82:0/3525090821 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe3cc054ef0 con 0x7fe3d80ffd50 2024-06-06T07:43:16.705 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.705+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe3d8107740 con 0x7fe3d80ffd50 2024-06-06T07:43:16.705 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.705+0000 7fe3d57fa640 1 -- 172.21.15.82:0/3525090821 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe3cc01e0d0 con 0x7fe3d80ffd50 2024-06-06T07:43:16.705 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:16.705 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:16.707 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe3c006aef0 msgr2=0x7fe3c006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:16.707 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe3c006aef0 0x7fe3c006d3b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009880 tx=0x7fe3c8000bc0 comp rx=0 tx=0).stop 2024-06-06T07:43:16.707 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ffd50 msgr2=0x7fe3d81967a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:16.707 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ffd50 0x7fe3d81967a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c8f0 tx=0x7fe3cc00cdb0 comp rx=0 tx=0).stop 2024-06-06T07:43:16.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 shutdown_connections 2024-06-06T07:43:16.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe3c006aef0 0x7fe3c006d3b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:16.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3d80ffd50 0x7fe3d81967a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:16.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 --2- 172.21.15.82:0/3525090821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3d80ff3a0 0x7fe3d8196260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:16.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 >> 172.21.15.82:0/3525090821 conn(0x7fe3d80fac50 msgr2=0x7fe3d8109590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:16.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 shutdown_connections 2024-06-06T07:43:16.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:16.707+0000 7fe3de29f640 1 -- 172.21.15.82:0/3525090821 wait complete. 2024-06-06T07:43:16.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:16 smithi082 ceph-mon[29354]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:17.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:16 smithi184 ceph-mon[37288]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:17.719 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:17 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3525090821' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:18.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:17 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3525090821' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:19.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:18 smithi082 ceph-mon[29354]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:18 smithi184 ceph-mon[37288]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:20.467 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:20.687 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:20.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:20 smithi082 ceph-mon[29354]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:21.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:20 smithi184 ceph-mon[37288]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:22.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 -- 172.21.15.82:0/395971635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 msgr2=0x7fae541017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:22.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 --2- 172.21.15.82:0/395971635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 0x7fae541017a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fae44009c20 tx=0x7fae4402aa00 comp rx=0 tx=0).stop 2024-06-06T07:43:22.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 -- 172.21.15.82:0/395971635 shutdown_connections 2024-06-06T07:43:22.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 --2- 172.21.15.82:0/395971635 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae54101ce0 0x7fae5410a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:22.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 --2- 172.21.15.82:0/395971635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 0x7fae541017a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:22.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 -- 172.21.15.82:0/395971635 >> 172.21.15.82:0/395971635 conn(0x7fae54075e10 msgr2=0x7fae540fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:22.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 -- 172.21.15.82:0/395971635 shutdown_connections 2024-06-06T07:43:22.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 -- 172.21.15.82:0/395971635 wait complete. 2024-06-06T07:43:22.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 Processor -- start 2024-06-06T07:43:22.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.435+0000 7fae593fb640 1 -- start start 2024-06-06T07:43:22.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae593fb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 0x7fae54196240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:22.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae593fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae54101ce0 0x7fae54196780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:22.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae593fb640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae54196e10 con 0x7fae541013c0 2024-06-06T07:43:22.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae593fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5419ab30 con 0x7fae54101ce0 2024-06-06T07:43:22.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae52ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 0x7fae54196240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:22.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae52ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 0x7fae54196240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:37272/0 (socket says 172.21.15.82:37272) 2024-06-06T07:43:22.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae52ffd640 1 -- 172.21.15.82:0/2337887658 learned_addr learned my addr 172.21.15.82:0/2337887658 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:22.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae527fc640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae54101ce0 0x7fae54196780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:22.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae52ffd640 1 -- 172.21.15.82:0/2337887658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae54101ce0 msgr2=0x7fae54196780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:22.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae52ffd640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae54101ce0 0x7fae54196780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:22.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae52ffd640 1 -- 172.21.15.82:0/2337887658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae44009900 con 0x7fae541013c0 2024-06-06T07:43:22.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae52ffd640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 0x7fae54196240 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fae4402af20 tx=0x7fae4402f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:22.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae4bfff640 1 -- 172.21.15.82:0/2337887658 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae4403e070 con 0x7fae541013c0 2024-06-06T07:43:22.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae4bfff640 1 -- 172.21.15.82:0/2337887658 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae4402e870 con 0x7fae541013c0 2024-06-06T07:43:22.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5419adb0 con 0x7fae541013c0 2024-06-06T07:43:22.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae4bfff640 1 -- 172.21.15.82:0/2337887658 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae44004830 con 0x7fae541013c0 2024-06-06T07:43:22.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.436+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5419b2a0 con 0x7fae541013c0 2024-06-06T07:43:22.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.437+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae54068510 con 0x7fae541013c0 2024-06-06T07:43:22.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.438+0000 7fae4bfff640 1 -- 172.21.15.82:0/2337887658 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fae4400a040 con 0x7fae541013c0 2024-06-06T07:43:22.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.438+0000 7fae4bfff640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae4006aef0 0x7fae4006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:22.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.438+0000 7fae4bfff640 1 -- 172.21.15.82:0/2337887658 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fae440aada0 con 0x7fae541013c0 2024-06-06T07:43:22.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.438+0000 7fae527fc640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae4006aef0 0x7fae4006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:22.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.439+0000 7fae527fc640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae4006aef0 0x7fae4006d3b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0058c0 tx=0x7fae3c005850 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:22.446 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.443+0000 7fae4bfff640 1 -- 172.21.15.82:0/2337887658 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fae44076380 con 0x7fae541013c0 2024-06-06T07:43:22.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.655+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fae5404e210 con 0x7fae541013c0 2024-06-06T07:43:22.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.655+0000 7fae4bfff640 1 -- 172.21.15.82:0/2337887658 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fae440799a0 con 0x7fae541013c0 2024-06-06T07:43:22.656 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:22.656 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:22.657 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae4006aef0 msgr2=0x7fae4006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae4006aef0 0x7fae4006d3b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0058c0 tx=0x7fae3c005850 comp rx=0 tx=0).stop 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 msgr2=0x7fae54196240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 0x7fae54196240 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fae4402af20 tx=0x7fae4402f870 comp rx=0 tx=0).stop 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 shutdown_connections 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae4006aef0 0x7fae4006d3b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae54101ce0 0x7fae54196780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 --2- 172.21.15.82:0/2337887658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae541013c0 0x7fae54196240 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 >> 172.21.15.82:0/2337887658 conn(0x7fae54075e10 msgr2=0x7fae541026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 shutdown_connections 2024-06-06T07:43:22.658 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:22.657+0000 7fae593fb640 1 -- 172.21.15.82:0/2337887658 wait complete. 2024-06-06T07:43:23.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:22 smithi082 ceph-mon[29354]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:23.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:43:23.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:22 smithi184 ceph-mon[37288]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:23.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:43:23.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:23 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2337887658' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:24.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:23 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2337887658' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:25.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:24 smithi082 ceph-mon[29354]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:25.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:24 smithi184 ceph-mon[37288]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:26.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:26.667 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:26.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:26 smithi082 ceph-mon[29354]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:27.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:26 smithi184 ceph-mon[37288]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:28.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.305+0000 7f542d942640 1 -- 172.21.15.82:0/3083238010 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ffd10 msgr2=0x7f5428103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:28.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.305+0000 7f542d942640 1 --2- 172.21.15.82:0/3083238010 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ffd10 0x7f5428103d60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f541c009880 tx=0x7f541c02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.305+0000 7f542d942640 1 -- 172.21.15.82:0/3083238010 shutdown_connections 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.305+0000 7f542d942640 1 --2- 172.21.15.82:0/3083238010 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ffd10 0x7f5428103d60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.305+0000 7f542d942640 1 --2- 172.21.15.82:0/3083238010 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54280ff360 0x7f54280ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.305+0000 7f542d942640 1 -- 172.21.15.82:0/3083238010 >> 172.21.15.82:0/3083238010 conn(0x7f5428075e10 msgr2=0x7f54280fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.305+0000 7f542d942640 1 -- 172.21.15.82:0/3083238010 shutdown_connections 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.305+0000 7f542d942640 1 -- 172.21.15.82:0/3083238010 wait complete. 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.306+0000 7f542d942640 1 Processor -- start 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.306+0000 7f542d942640 1 -- start start 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.306+0000 7f542d942640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ff360 0x7f5428198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.306+0000 7f542d942640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54280ffd10 0x7f5428198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.306+0000 7f542d942640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5428199010 con 0x7f54280ff360 2024-06-06T07:43:28.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.306+0000 7f542d942640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542819cd30 con 0x7f54280ffd10 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.306+0000 7f5426ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ff360 0x7f5428198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.306+0000 7f5426ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ff360 0x7f5428198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:48278/0 (socket says 172.21.15.82:48278) 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f5426ffd640 1 -- 172.21.15.82:0/942342538 learned_addr learned my addr 172.21.15.82:0/942342538 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f5426ffd640 1 -- 172.21.15.82:0/942342538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54280ffd10 msgr2=0x7f5428198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f5426ffd640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54280ffd10 0x7f5428198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f5426ffd640 1 -- 172.21.15.82:0/942342538 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c009560 con 0x7f54280ff360 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f5426ffd640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ff360 0x7f5428198440 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f541800ea00 tx=0x7f541800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f542c940640 1 -- 172.21.15.82:0/942342538 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f541800cd50 con 0x7f54280ff360 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542819d010 con 0x7f54280ff360 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f542c940640 1 -- 172.21.15.82:0/942342538 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5418005020 con 0x7f54280ff360 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f542c940640 1 -- 172.21.15.82:0/942342538 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54180116a0 con 0x7f54280ff360 2024-06-06T07:43:28.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.307+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542819d560 con 0x7f54280ff360 2024-06-06T07:43:28.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.308+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5428100840 con 0x7f54280ff360 2024-06-06T07:43:28.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.308+0000 7f542c940640 1 -- 172.21.15.82:0/942342538 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f5418005190 con 0x7f54280ff360 2024-06-06T07:43:28.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.309+0000 7f542c940640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f541006aef0 0x7f541006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:28.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.309+0000 7f542c940640 1 -- 172.21.15.82:0/942342538 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f5418089fd0 con 0x7f54280ff360 2024-06-06T07:43:28.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.309+0000 7f54267fc640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f541006aef0 0x7f541006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:28.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.309+0000 7f54267fc640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f541006aef0 0x7f541006d3b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f54281999a0 tx=0x7f541c000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:28.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.313+0000 7f542c940640 1 -- 172.21.15.82:0/942342538 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f54180555b0 con 0x7f54280ff360 2024-06-06T07:43:28.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.522+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f542804e210 con 0x7f54280ff360 2024-06-06T07:43:28.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.522+0000 7f542c940640 1 -- 172.21.15.82:0/942342538 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5418058bd0 con 0x7f54280ff360 2024-06-06T07:43:28.522 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:28.522 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:28.524 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f541006aef0 msgr2=0x7f541006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:28.524 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f541006aef0 0x7f541006d3b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f54281999a0 tx=0x7f541c000bc0 comp rx=0 tx=0).stop 2024-06-06T07:43:28.524 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ff360 msgr2=0x7f5428198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:28.524 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ff360 0x7f5428198440 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f541800ea00 tx=0x7f541800eec0 comp rx=0 tx=0).stop 2024-06-06T07:43:28.524 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 shutdown_connections 2024-06-06T07:43:28.524 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f541006aef0 0x7f541006d3b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:28.524 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54280ffd10 0x7f5428198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:28.524 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 --2- 172.21.15.82:0/942342538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54280ff360 0x7f5428198440 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:28.525 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 >> 172.21.15.82:0/942342538 conn(0x7f5428075e10 msgr2=0x7f5428102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:28.525 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 shutdown_connections 2024-06-06T07:43:28.525 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:28.524+0000 7f542d942640 1 -- 172.21.15.82:0/942342538 wait complete. 2024-06-06T07:43:28.881 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:28 smithi082 ceph-mon[29354]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:28.881 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:28 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/942342538' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:29.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:28 smithi184 ceph-mon[37288]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:29.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:28 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/942342538' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:30.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:29 smithi082 ceph-mon[29354]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:30.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:29 smithi184 ceph-mon[37288]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:32.197 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:32.416 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:32.842 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:32 smithi082 ceph-mon[29354]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:33.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:32 smithi184 ceph-mon[37288]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:33.962 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.960+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3467095980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 msgr2=0x7f7d2c100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:33.962 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.960+0000 7f7d32ae7640 1 --2- 172.21.15.82:0/3467095980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 0x7f7d2c100860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009880 tx=0x7f7d1c02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:43:33.962 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.960+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3467095980 shutdown_connections 2024-06-06T07:43:33.962 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.960+0000 7f7d32ae7640 1 --2- 172.21.15.82:0/3467095980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d2c100e30 0x7f7d2c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:33.963 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.960+0000 7f7d32ae7640 1 --2- 172.21.15.82:0/3467095980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 0x7f7d2c100860 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:33.963 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.960+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3467095980 >> 172.21.15.82:0/3467095980 conn(0x7f7d2c0fbd30 msgr2=0x7f7d2c0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:33.963 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.960+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3467095980 shutdown_connections 2024-06-06T07:43:33.963 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.960+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3467095980 wait complete. 2024-06-06T07:43:33.963 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.960+0000 7f7d32ae7640 1 Processor -- start 2024-06-06T07:43:33.963 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d32ae7640 1 -- start start 2024-06-06T07:43:33.963 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d32ae7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 0x7f7d2c196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d32ae7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d2c100e30 0x7f7d2c196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d32ae7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c196ef0 con 0x7f7d2c100480 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d32ae7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c19ac10 con 0x7f7d2c100e30 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d30883640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 0x7f7d2c196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d30883640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 0x7f7d2c196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:48298/0 (socket says 172.21.15.82:48298) 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d30883640 1 -- 172.21.15.82:0/3779071337 learned_addr learned my addr 172.21.15.82:0/3779071337 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d2bfff640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d2c100e30 0x7f7d2c196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d30883640 1 -- 172.21.15.82:0/3779071337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d2c100e30 msgr2=0x7f7d2c196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d30883640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d2c100e30 0x7f7d2c196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d30883640 1 -- 172.21.15.82:0/3779071337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c009560 con 0x7f7d2c100480 2024-06-06T07:43:33.964 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d2bfff640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d2c100e30 0x7f7d2c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T07:43:33.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.961+0000 7f7d30883640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 0x7f7d2c196320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c006010 tx=0x7f7d1c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:33.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.962+0000 7f7d29ffb640 1 -- 172.21.15.82:0/3779071337 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d1c004aa0 con 0x7f7d2c100480 2024-06-06T07:43:33.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.962+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c19ae90 con 0x7f7d2c100480 2024-06-06T07:43:33.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.962+0000 7f7d29ffb640 1 -- 172.21.15.82:0/3779071337 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d1c004c00 con 0x7f7d2c100480 2024-06-06T07:43:33.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.962+0000 7f7d29ffb640 1 -- 172.21.15.82:0/3779071337 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d1c003a20 con 0x7f7d2c100480 2024-06-06T07:43:33.965 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.962+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c19b380 con 0x7f7d2c100480 2024-06-06T07:43:33.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.963+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c101960 con 0x7f7d2c100480 2024-06-06T07:43:33.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.963+0000 7f7d29ffb640 1 -- 172.21.15.82:0/3779071337 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7d1c031c90 con 0x7f7d2c100480 2024-06-06T07:43:33.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.964+0000 7f7d29ffb640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f7d1806af00 0x7f7d1806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:33.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.964+0000 7f7d29ffb640 1 -- 172.21.15.82:0/3779071337 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7d1c0aae20 con 0x7f7d2c100480 2024-06-06T07:43:33.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.964+0000 7f7d2bfff640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f7d1806af00 0x7f7d1806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:33.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.964+0000 7f7d2bfff640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f7d1806af00 0x7f7d1806d3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c197880 tx=0x7f7d14008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:33.968 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:33.968+0000 7f7d29ffb640 1 -- 172.21.15.82:0/3779071337 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7d1c076400 con 0x7f7d2c100480 2024-06-06T07:43:34.176 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.176+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7d2c04e210 con 0x7f7d2c100480 2024-06-06T07:43:34.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.177+0000 7f7d29ffb640 1 -- 172.21.15.82:0/3779071337 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7d1c079a20 con 0x7f7d2c100480 2024-06-06T07:43:34.177 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:34.177 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.178+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f7d1806af00 msgr2=0x7f7d1806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.178+0000 7f7d32ae7640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f7d1806af00 0x7f7d1806d3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c197880 tx=0x7f7d14008040 comp rx=0 tx=0).stop 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.178+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 msgr2=0x7f7d2c196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.178+0000 7f7d32ae7640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 0x7f7d2c196320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c006010 tx=0x7f7d1c0055a0 comp rx=0 tx=0).stop 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.179+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 shutdown_connections 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.179+0000 7f7d32ae7640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f7d1806af00 0x7f7d1806d3c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.179+0000 7f7d32ae7640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d2c100e30 0x7f7d2c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.179+0000 7f7d32ae7640 1 --2- 172.21.15.82:0/3779071337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d2c100480 0x7f7d2c196320 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.179+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 >> 172.21.15.82:0/3779071337 conn(0x7f7d2c0fbd30 msgr2=0x7f7d2c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.179+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 shutdown_connections 2024-06-06T07:43:34.180 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:34.179+0000 7f7d32ae7640 1 -- 172.21.15.82:0/3779071337 wait complete. 2024-06-06T07:43:34.851 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:34 smithi082 ceph-mon[29354]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:34.852 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:34 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3779071337' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:34.896 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:34 smithi184 ceph-mon[37288]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:34.896 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:34 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3779071337' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:36.954 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:36 smithi184 ceph-mon[37288]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:37.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:36 smithi082 ceph-mon[29354]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:37.873 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:37.903 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:43:38.090 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:38.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:43:38.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:38 smithi082 ceph-mon[29354]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:39.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:38 smithi184 ceph-mon[37288]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:39.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.881+0000 7f04b5151640 1 -- 172.21.15.82:0/3056425563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b0104950 msgr2=0x7f04b0104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:39.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.881+0000 7f04b5151640 1 --2- 172.21.15.82:0/3056425563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b0104950 0x7f04b0104d30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0498009880 tx=0x7f049802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:43:39.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.881+0000 7f04b5151640 1 -- 172.21.15.82:0/3056425563 shutdown_connections 2024-06-06T07:43:39.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.881+0000 7f04b5151640 1 --2- 172.21.15.82:0/3056425563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04b00fe950 0x7f04b00fedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:39.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.881+0000 7f04b5151640 1 --2- 172.21.15.82:0/3056425563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b0104950 0x7f04b0104d30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:39.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.881+0000 7f04b5151640 1 -- 172.21.15.82:0/3056425563 >> 172.21.15.82:0/3056425563 conn(0x7f04b00755f0 msgr2=0x7f04b00fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:39.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.881+0000 7f04b5151640 1 -- 172.21.15.82:0/3056425563 shutdown_connections 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.881+0000 7f04b5151640 1 -- 172.21.15.82:0/3056425563 wait complete. 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.882+0000 7f04b5151640 1 Processor -- start 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.882+0000 7f04b5151640 1 -- start start 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.882+0000 7f04b5151640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b00fe950 0x7f04b0197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.882+0000 7f04b5151640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04b0104950 0x7f04b01975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.882+0000 7f04b5151640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0197af0 con 0x7f04b00fe950 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.882+0000 7f04b5151640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0197c60 con 0x7f04b0104950 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.882+0000 7f04aed9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b00fe950 0x7f04b0197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.882+0000 7f04aed9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b00fe950 0x7f04b0197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:54032/0 (socket says 172.21.15.82:54032) 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.882+0000 7f04aed9d640 1 -- 172.21.15.82:0/2590006315 learned_addr learned my addr 172.21.15.82:0/2590006315 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.883+0000 7f04aed9d640 1 -- 172.21.15.82:0/2590006315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04b0104950 msgr2=0x7f04b01975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:39.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.883+0000 7f04aed9d640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04b0104950 0x7f04b01975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.883+0000 7f04aed9d640 1 -- 172.21.15.82:0/2590006315 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0498009560 con 0x7f04b00fe950 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.883+0000 7f04aed9d640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b00fe950 0x7f04b0197070 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f049802a9c0 tx=0x7f04980055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.883+0000 7f04a7fff640 1 -- 172.21.15.82:0/2590006315 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0498004aa0 con 0x7f04b00fe950 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.883+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b0197ee0 con 0x7f04b00fe950 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.883+0000 7f04a7fff640 1 -- 172.21.15.82:0/2590006315 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0498004c00 con 0x7f04b00fe950 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.883+0000 7f04a7fff640 1 -- 172.21.15.82:0/2590006315 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0498003ab0 con 0x7f04b00fe950 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.883+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b01a03f0 con 0x7f04b00fe950 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.884+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b0100090 con 0x7f04b00fe950 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.884+0000 7f04a7fff640 1 -- 172.21.15.82:0/2590006315 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0498003ce0 con 0x7f04b00fe950 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.885+0000 7f04a7fff640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f049c06aef0 0x7f049c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.885+0000 7f04a7fff640 1 -- 172.21.15.82:0/2590006315 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f04980aabb0 con 0x7f04b00fe950 2024-06-06T07:43:39.898 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.885+0000 7f04ae59c640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f049c06aef0 0x7f049c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:39.899 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.885+0000 7f04ae59c640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f049c06aef0 0x7f049c06d3b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f04b0101fb0 tx=0x7f04a0005c70 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:39.899 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:39.889+0000 7f04a7fff640 1 -- 172.21.15.82:0/2590006315 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f04980b1080 con 0x7f04b00fe950 2024-06-06T07:43:40.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.099+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f04b0197ee0 con 0x7f04b00fe950 2024-06-06T07:43:40.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.099+0000 7f04a7fff640 1 -- 172.21.15.82:0/2590006315 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f04b0197ee0 con 0x7f04b00fe950 2024-06-06T07:43:40.100 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:40.100 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:40.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.101+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f049c06aef0 msgr2=0x7f049c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.101+0000 7f04b5151640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f049c06aef0 0x7f049c06d3b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f04b0101fb0 tx=0x7f04a0005c70 comp rx=0 tx=0).stop 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.101+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b00fe950 msgr2=0x7f04b0197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.101+0000 7f04b5151640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b00fe950 0x7f04b0197070 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f049802a9c0 tx=0x7f04980055a0 comp rx=0 tx=0).stop 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.101+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 shutdown_connections 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.101+0000 7f04b5151640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f049c06aef0 0x7f049c06d3b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.101+0000 7f04b5151640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04b0104950 0x7f04b01975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.101+0000 7f04b5151640 1 --2- 172.21.15.82:0/2590006315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f04b00fe950 0x7f04b0197070 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.102+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 >> 172.21.15.82:0/2590006315 conn(0x7f04b00755f0 msgr2=0x7f04b00fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.102+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 shutdown_connections 2024-06-06T07:43:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:40.102+0000 7f04b5151640 1 -- 172.21.15.82:0/2590006315 wait complete. 2024-06-06T07:43:40.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:40 smithi082 ceph-mon[29354]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:40.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:40 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2590006315' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:41.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:40 smithi184 ceph-mon[37288]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:41.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:40 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2590006315' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:42.768 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:42 smithi082 ceph-mon[29354]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:43.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:42 smithi184 ceph-mon[37288]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:43.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:44.034 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:44.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:44 smithi082 ceph-mon[29354]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:45.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:44 smithi184 ceph-mon[37288]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:45.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.646+0000 7f330ac52640 1 -- 172.21.15.82:0/183844016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ff3a0 msgr2=0x7f33040ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:45.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.646+0000 7f330ac52640 1 --2- 172.21.15.82:0/183844016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ff3a0 0x7f33040ff780 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009880 tx=0x7f32f402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:43:45.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.646+0000 7f330ac52640 1 -- 172.21.15.82:0/183844016 shutdown_connections 2024-06-06T07:43:45.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.646+0000 7f330ac52640 1 --2- 172.21.15.82:0/183844016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33040ffd50 0x7f3304103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:45.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.646+0000 7f330ac52640 1 --2- 172.21.15.82:0/183844016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ff3a0 0x7f33040ff780 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:45.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.646+0000 7f330ac52640 1 -- 172.21.15.82:0/183844016 >> 172.21.15.82:0/183844016 conn(0x7f33040fac50 msgr2=0x7f33040fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:45.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.646+0000 7f330ac52640 1 -- 172.21.15.82:0/183844016 shutdown_connections 2024-06-06T07:43:45.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.646+0000 7f330ac52640 1 -- 172.21.15.82:0/183844016 wait complete. 2024-06-06T07:43:45.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f330ac52640 1 Processor -- start 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f330ac52640 1 -- start start 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f330ac52640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33040ff3a0 0x7f3304196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f330ac52640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ffd50 0x7f3304196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f330ac52640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3304196f10 con 0x7f33040ffd50 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f330ac52640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330419ac30 con 0x7f33040ff3a0 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f3303fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ffd50 0x7f3304196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f3303fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ffd50 0x7f3304196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:34210/0 (socket says 172.21.15.82:34210) 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f3303fff640 1 -- 172.21.15.82:0/2930959414 learned_addr learned my addr 172.21.15.82:0/2930959414 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f3303fff640 1 -- 172.21.15.82:0/2930959414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33040ff3a0 msgr2=0x7f3304196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f3303fff640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33040ff3a0 0x7f3304196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f3303fff640 1 -- 172.21.15.82:0/2930959414 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f4009560 con 0x7f33040ffd50 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f3303fff640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ffd50 0x7f3304196880 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00b3d0 tx=0x7f32ec00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:45.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f3301ffb640 1 -- 172.21.15.82:0/2930959414 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32ec011800 con 0x7f33040ffd50 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f3301ffb640 1 -- 172.21.15.82:0/2930959414 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32ec011e20 con 0x7f33040ffd50 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.647+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330419af10 con 0x7f33040ffd50 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.648+0000 7f3301ffb640 1 -- 172.21.15.82:0/2930959414 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32ec00f2e0 con 0x7f33040ffd50 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.648+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330419b460 con 0x7f33040ffd50 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.649+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3304100880 con 0x7f33040ffd50 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.649+0000 7f3301ffb640 1 -- 172.21.15.82:0/2930959414 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f32ec010380 con 0x7f33040ffd50 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.649+0000 7f3301ffb640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f32f006aef0 0x7f32f006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.650+0000 7f3301ffb640 1 -- 172.21.15.82:0/2930959414 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f32ec089930 con 0x7f33040ffd50 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.650+0000 7f33089ee640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f32f006aef0 0x7f32f006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:45.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.650+0000 7f33089ee640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f32f006aef0 0x7f32f006d3b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009880 tx=0x7f32f4000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:45.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.654+0000 7f3301ffb640 1 -- 172.21.15.82:0/2930959414 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f32ec054f10 con 0x7f33040ffd50 2024-06-06T07:43:45.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.865+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f330404e210 con 0x7f33040ffd50 2024-06-06T07:43:45.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.866+0000 7f3301ffb640 1 -- 172.21.15.82:0/2930959414 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f32ec058530 con 0x7f33040ffd50 2024-06-06T07:43:45.866 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:45.867 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:45.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f32f006aef0 msgr2=0x7f32f006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:45.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f32f006aef0 0x7f32f006d3b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009880 tx=0x7f32f4000bc0 comp rx=0 tx=0).stop 2024-06-06T07:43:45.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ffd50 msgr2=0x7f3304196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:45.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ffd50 0x7f3304196880 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00b3d0 tx=0x7f32ec00b890 comp rx=0 tx=0).stop 2024-06-06T07:43:45.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 shutdown_connections 2024-06-06T07:43:45.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f32f006aef0 0x7f32f006d3b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:45.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f33040ffd50 0x7f3304196880 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:45.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 --2- 172.21.15.82:0/2930959414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33040ff3a0 0x7f3304196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:45.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 >> 172.21.15.82:0/2930959414 conn(0x7f33040fac50 msgr2=0x7f3304068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:45.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 shutdown_connections 2024-06-06T07:43:45.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:45.868+0000 7f330ac52640 1 -- 172.21.15.82:0/2930959414 wait complete. 2024-06-06T07:43:46.990 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:46 smithi082 ceph-mon[29354]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:46.990 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:46 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2930959414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:47.073 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:46 smithi184 ceph-mon[37288]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:47.074 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:46 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2930959414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:49.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:48 smithi082 ceph-mon[29354]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:49.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:48 smithi184 ceph-mon[37288]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:49.736 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:49.953 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:50.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:50 smithi082 ceph-mon[29354]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:51.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:50 smithi184 ceph-mon[37288]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:51.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 -- 172.21.15.82:0/2873204564 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ffd60 msgr2=0x7f91a0103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 --2- 172.21.15.82:0/2873204564 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ffd60 0x7f91a0103db0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9194009880 tx=0x7f919402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 -- 172.21.15.82:0/2873204564 shutdown_connections 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 --2- 172.21.15.82:0/2873204564 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ffd60 0x7f91a0103db0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 --2- 172.21.15.82:0/2873204564 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91a00ff3b0 0x7f91a00ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 -- 172.21.15.82:0/2873204564 >> 172.21.15.82:0/2873204564 conn(0x7f91a00fac80 msgr2=0x7f91a00fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 -- 172.21.15.82:0/2873204564 shutdown_connections 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 -- 172.21.15.82:0/2873204564 wait complete. 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 Processor -- start 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.641+0000 7f91a4821640 1 -- start start 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f91a4821640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ff3b0 0x7f91a01962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f91a4821640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91a00ffd60 0x7f91a01967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:51.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f91a4821640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a0196e70 con 0x7f91a00ff3b0 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f91a4821640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a019ab90 con 0x7f91a00ffd60 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f919e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ff3b0 0x7f91a01962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f919e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ff3b0 0x7f91a01962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:34234/0 (socket says 172.21.15.82:34234) 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f919e59c640 1 -- 172.21.15.82:0/2199501530 learned_addr learned my addr 172.21.15.82:0/2199501530 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f919dd9b640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91a00ffd60 0x7f91a01967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f919e59c640 1 -- 172.21.15.82:0/2199501530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91a00ffd60 msgr2=0x7f91a01967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f919e59c640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91a00ffd60 0x7f91a01967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f919e59c640 1 -- 172.21.15.82:0/2199501530 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9194009560 con 0x7f91a00ff3b0 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f919e59c640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ff3b0 0x7f91a01962a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f919000d6f0 tx=0x7f919000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f918f7fe640 1 -- 172.21.15.82:0/2199501530 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9190009a30 con 0x7f91a00ff3b0 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a019ae70 con 0x7f91a00ff3b0 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f918f7fe640 1 -- 172.21.15.82:0/2199501530 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91900054e0 con 0x7f91a00ff3b0 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f918f7fe640 1 -- 172.21.15.82:0/2199501530 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9190010460 con 0x7f91a00ff3b0 2024-06-06T07:43:51.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.642+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a019b390 con 0x7f91a00ff3b0 2024-06-06T07:43:51.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.643+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a0100890 con 0x7f91a00ff3b0 2024-06-06T07:43:51.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.644+0000 7f918f7fe640 1 -- 172.21.15.82:0/2199501530 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f9190009b90 con 0x7f91a00ff3b0 2024-06-06T07:43:51.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.644+0000 7f918f7fe640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f918806aef0 0x7f918806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:51.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.644+0000 7f918f7fe640 1 -- 172.21.15.82:0/2199501530 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f91900897a0 con 0x7f91a00ff3b0 2024-06-06T07:43:51.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.644+0000 7f919dd9b640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f918806aef0 0x7f918806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:51.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.645+0000 7f919dd9b640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f918806aef0 0x7f918806d3b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9194009230 tx=0x7f9194005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:51.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.649+0000 7f918f7fe640 1 -- 172.21.15.82:0/2199501530 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9190054d80 con 0x7f91a00ff3b0 2024-06-06T07:43:51.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.861+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f91a004e210 con 0x7f91a00ff3b0 2024-06-06T07:43:51.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.862+0000 7f918f7fe640 1 -- 172.21.15.82:0/2199501530 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f91900200d0 con 0x7f91a00ff3b0 2024-06-06T07:43:51.863 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:51.863 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f918806aef0 msgr2=0x7f918806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f918806aef0 0x7f918806d3b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9194009230 tx=0x7f9194005a20 comp rx=0 tx=0).stop 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ff3b0 msgr2=0x7f91a01962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ff3b0 0x7f91a01962a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f919000d6f0 tx=0x7f919000dbb0 comp rx=0 tx=0).stop 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 shutdown_connections 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f918806aef0 0x7f918806d3b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91a00ffd60 0x7f91a01967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 --2- 172.21.15.82:0/2199501530 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91a00ff3b0 0x7f91a01962a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 >> 172.21.15.82:0/2199501530 conn(0x7f91a00fac80 msgr2=0x7f91a0068d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:51.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 shutdown_connections 2024-06-06T07:43:51.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:51.864+0000 7f91a4821640 1 -- 172.21.15.82:0/2199501530 wait complete. 2024-06-06T07:43:53.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:52 smithi082 ceph-mon[29354]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:53.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:52 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2199501530' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:53.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:43:53.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:52 smithi184 ceph-mon[37288]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:52 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2199501530' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:43:55.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:54 smithi082 ceph-mon[29354]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:55.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:54 smithi184 ceph-mon[37288]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:55.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:43:55.864 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:43:57.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:56 smithi082 ceph-mon[29354]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:57.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:56 smithi184 ceph-mon[37288]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:57.577 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.563+0000 7f812a048640 1 -- 172.21.15.82:0/182507858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100e30 msgr2=0x7f8124104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:57.577 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.563+0000 7f812a048640 1 --2- 172.21.15.82:0/182507858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100e30 0x7f8124104e80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f810c009880 tx=0x7f810c02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:43:57.577 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.563+0000 7f812a048640 1 -- 172.21.15.82:0/182507858 shutdown_connections 2024-06-06T07:43:57.577 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.563+0000 7f812a048640 1 --2- 172.21.15.82:0/182507858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100e30 0x7f8124104e80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:57.577 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.563+0000 7f812a048640 1 --2- 172.21.15.82:0/182507858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8124100480 0x7f8124100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:57.577 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.563+0000 7f812a048640 1 -- 172.21.15.82:0/182507858 >> 172.21.15.82:0/182507858 conn(0x7f81240fbd30 msgr2=0x7f81240fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:57.577 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.563+0000 7f812a048640 1 -- 172.21.15.82:0/182507858 shutdown_connections 2024-06-06T07:43:57.577 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.563+0000 7f812a048640 1 -- 172.21.15.82:0/182507858 wait complete. 2024-06-06T07:43:57.577 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f812a048640 1 Processor -- start 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f812a048640 1 -- start start 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f812a048640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100480 0x7f8124196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f812a048640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8124100e30 0x7f8124196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f812a048640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8124196ed0 con 0x7f8124100480 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f812a048640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812419abf0 con 0x7f8124100e30 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f81237fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100480 0x7f8124196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f81237fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100480 0x7f8124196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:53752/0 (socket says 172.21.15.82:53752) 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f81237fe640 1 -- 172.21.15.82:0/2820898965 learned_addr learned my addr 172.21.15.82:0/2820898965 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f811bfff640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8124100e30 0x7f8124196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f81237fe640 1 -- 172.21.15.82:0/2820898965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8124100e30 msgr2=0x7f8124196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f81237fe640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8124100e30 0x7f8124196840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.564+0000 7f81237fe640 1 -- 172.21.15.82:0/2820898965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c009560 con 0x7f8124100480 2024-06-06T07:43:57.578 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.565+0000 7f81237fe640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100480 0x7f8124196300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f811400b420 tx=0x7f811400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.565+0000 7f81217fa640 1 -- 172.21.15.82:0/2820898965 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8114011800 con 0x7f8124100480 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.565+0000 7f81217fa640 1 -- 172.21.15.82:0/2820898965 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8114011e20 con 0x7f8124100480 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.565+0000 7f81217fa640 1 -- 172.21.15.82:0/2820898965 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f811400f2e0 con 0x7f8124100480 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.565+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812419aed0 con 0x7f8124100480 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.565+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812419b3f0 con 0x7f8124100480 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.566+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8124101960 con 0x7f8124100480 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.566+0000 7f81217fa640 1 -- 172.21.15.82:0/2820898965 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f8114010380 con 0x7f8124100480 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.567+0000 7f81217fa640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f811006aef0 0x7f811006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.567+0000 7f811bfff640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f811006aef0 0x7f811006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.567+0000 7f81217fa640 1 -- 172.21.15.82:0/2820898965 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f81140894d0 con 0x7f8124100480 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.567+0000 7f811bfff640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f811006aef0 0x7f811006d3b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f810c009230 tx=0x7f810c005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:43:57.579 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.571+0000 7f81217fa640 1 -- 172.21.15.82:0/2820898965 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8114054b30 con 0x7f8124100480 2024-06-06T07:43:57.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.786+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f812404e210 con 0x7f8124100480 2024-06-06T07:43:57.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.786+0000 7f81217fa640 1 -- 172.21.15.82:0/2820898965 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8114058150 con 0x7f8124100480 2024-06-06T07:43:57.786 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:43:57.786 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.788+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f811006aef0 msgr2=0x7f811006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.788+0000 7f812a048640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f811006aef0 0x7f811006d3b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f810c009230 tx=0x7f810c005a20 comp rx=0 tx=0).stop 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.788+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100480 msgr2=0x7f8124196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.788+0000 7f812a048640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100480 0x7f8124196300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f811400b420 tx=0x7f811400b8e0 comp rx=0 tx=0).stop 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.788+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 shutdown_connections 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.788+0000 7f812a048640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f811006aef0 0x7f811006d3b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.788+0000 7f812a048640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8124100e30 0x7f8124196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.788+0000 7f812a048640 1 --2- 172.21.15.82:0/2820898965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8124100480 0x7f8124196300 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.788+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 >> 172.21.15.82:0/2820898965 conn(0x7f81240fbd30 msgr2=0x7f81240fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.789+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 shutdown_connections 2024-06-06T07:43:57.789 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:43:57.789+0000 7f812a048640 1 -- 172.21.15.82:0/2820898965 wait complete. 2024-06-06T07:43:58.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:58 smithi082 ceph-mon[29354]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:58.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:43:58 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2820898965' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:43:59.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:58 smithi184 ceph-mon[37288]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:43:59.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:43:58 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2820898965' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:01.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:00 smithi082 ceph-mon[29354]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:01.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:00 smithi184 ceph-mon[37288]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:01.496 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:01.719 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:02.962 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:02 smithi082 ceph-mon[29354]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:03.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:02 smithi184 ceph-mon[37288]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:03.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.444+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3590063630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ff3b0 msgr2=0x7fb6b80ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.444+0000 7fb6bcd53640 1 --2- 172.21.15.82:0/3590063630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ff3b0 0x7fb6b80ff790 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8009880 tx=0x7fb6a802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.444+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3590063630 shutdown_connections 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.444+0000 7fb6bcd53640 1 --2- 172.21.15.82:0/3590063630 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6b80ffd60 0x7fb6b8103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.444+0000 7fb6bcd53640 1 --2- 172.21.15.82:0/3590063630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ff3b0 0x7fb6b80ff790 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.444+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3590063630 >> 172.21.15.82:0/3590063630 conn(0x7fb6b80fac80 msgr2=0x7fb6b80fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.444+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3590063630 shutdown_connections 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.444+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3590063630 wait complete. 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.445+0000 7fb6bcd53640 1 Processor -- start 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.445+0000 7fb6bcd53640 1 -- start start 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.445+0000 7fb6bcd53640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6b80ff3b0 0x7fb6b8196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.445+0000 7fb6bcd53640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ffd60 0x7fb6b81968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:03.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.445+0000 7fb6bcd53640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b8196f60 con 0x7fb6b80ffd60 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.445+0000 7fb6bcd53640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b819ac30 con 0x7fb6b80ff3b0 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.445+0000 7fb6b5d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ffd60 0x7fb6b81968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.445+0000 7fb6b5d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ffd60 0x7fb6b81968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:53762/0 (socket says 172.21.15.82:53762) 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.445+0000 7fb6b5d9b640 1 -- 172.21.15.82:0/3248585972 learned_addr learned my addr 172.21.15.82:0/3248585972 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6b659c640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6b80ff3b0 0x7fb6b8196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6b5d9b640 1 -- 172.21.15.82:0/3248585972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6b80ff3b0 msgr2=0x7fb6b8196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6b5d9b640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6b80ff3b0 0x7fb6b8196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6b5d9b640 1 -- 172.21.15.82:0/3248585972 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a8009560 con 0x7fb6b80ffd60 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6b5d9b640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ffd60 0x7fb6b81968d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00b3d0 tx=0x7fb6ac00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6a77fe640 1 -- 172.21.15.82:0/3248585972 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6ac011800 con 0x7fb6b80ffd60 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6a77fe640 1 -- 172.21.15.82:0/3248585972 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6ac011e20 con 0x7fb6b80ffd60 2024-06-06T07:44:03.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b819af10 con 0x7fb6b80ffd60 2024-06-06T07:44:03.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6a77fe640 1 -- 172.21.15.82:0/3248585972 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6ac00f2e0 con 0x7fb6b80ffd60 2024-06-06T07:44:03.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.446+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b819b460 con 0x7fb6b80ffd60 2024-06-06T07:44:03.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.447+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b8100890 con 0x7fb6b80ffd60 2024-06-06T07:44:03.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.447+0000 7fb6a77fe640 1 -- 172.21.15.82:0/3248585972 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb6ac010380 con 0x7fb6b80ffd60 2024-06-06T07:44:03.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.448+0000 7fb6a77fe640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb6a006aef0 0x7fb6a006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:03.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.448+0000 7fb6b659c640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb6a006aef0 0x7fb6a006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:03.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.448+0000 7fb6a77fe640 1 -- 172.21.15.82:0/3248585972 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb6ac089930 con 0x7fb6b80ffd60 2024-06-06T07:44:03.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.448+0000 7fb6b659c640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb6a006aef0 0x7fb6a006d3b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8000c00 tx=0x7fb6a8004120 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:03.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.452+0000 7fb6a77fe640 1 -- 172.21.15.82:0/3248585972 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb6ac054f10 con 0x7fb6b80ffd60 2024-06-06T07:44:03.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.660+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb6b804e210 con 0x7fb6b80ffd60 2024-06-06T07:44:03.660 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.660+0000 7fb6a77fe640 1 -- 172.21.15.82:0/3248585972 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb6ac058530 con 0x7fb6b80ffd60 2024-06-06T07:44:03.660 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:03.660 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:03.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb6a006aef0 msgr2=0x7fb6a006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:03.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb6a006aef0 0x7fb6a006d3b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8000c00 tx=0x7fb6a8004120 comp rx=0 tx=0).stop 2024-06-06T07:44:03.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ffd60 msgr2=0x7fb6b81968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:03.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ffd60 0x7fb6b81968d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00b3d0 tx=0x7fb6ac00b890 comp rx=0 tx=0).stop 2024-06-06T07:44:03.662 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 shutdown_connections 2024-06-06T07:44:03.663 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb6a006aef0 0x7fb6a006d3b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:03.663 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6b80ffd60 0x7fb6b81968d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:03.663 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 --2- 172.21.15.82:0/3248585972 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6b80ff3b0 0x7fb6b8196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:03.663 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 >> 172.21.15.82:0/3248585972 conn(0x7fb6b80fac80 msgr2=0x7fb6b80689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:03.663 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 shutdown_connections 2024-06-06T07:44:03.663 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:03.662+0000 7fb6bcd53640 1 -- 172.21.15.82:0/3248585972 wait complete. 2024-06-06T07:44:03.969 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:03 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3248585972' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:04.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:03 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3248585972' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:04.994 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:04 smithi184 ceph-mon[37288]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:05.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:04 smithi082 ceph-mon[29354]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:07.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:06 smithi082 ceph-mon[29354]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:07.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:06 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:44:07.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:06 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:44:07.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:06 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:44:07.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:06 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:44:07.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:06 smithi184 ceph-mon[37288]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:06 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:44:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:06 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:44:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:06 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:44:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:06 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:44:07.360 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:07.575 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:08.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:44:08.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:44:08.917 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:08 smithi082 ceph-mon[29354]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:09.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:08 smithi184 ceph-mon[37288]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:09.128 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.123+0000 7fef694c4640 1 -- 172.21.15.82:0/3583513602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100470 msgr2=0x7fef64100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.123+0000 7fef694c4640 1 --2- 172.21.15.82:0/3583513602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100470 0x7fef64100850 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fef5c009c20 tx=0x7fef5c02a970 comp rx=0 tx=0).stop 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.123+0000 7fef694c4640 1 -- 172.21.15.82:0/3583513602 shutdown_connections 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.123+0000 7fef694c4640 1 --2- 172.21.15.82:0/3583513602 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef64100e20 0x7fef64104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.123+0000 7fef694c4640 1 --2- 172.21.15.82:0/3583513602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100470 0x7fef64100850 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.123+0000 7fef694c4640 1 -- 172.21.15.82:0/3583513602 >> 172.21.15.82:0/3583513602 conn(0x7fef640fbd20 msgr2=0x7fef640fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.123+0000 7fef694c4640 1 -- 172.21.15.82:0/3583513602 shutdown_connections 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.123+0000 7fef694c4640 1 -- 172.21.15.82:0/3583513602 wait complete. 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.123+0000 7fef694c4640 1 Processor -- start 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef694c4640 1 -- start start 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef694c4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef64100470 0x7fef64196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:09.129 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef694c4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100e20 0x7fef64196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef694c4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef64196e20 con 0x7fef64100e20 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef694c4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6419aaf0 con 0x7fef64100470 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef627fc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100e20 0x7fef64196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef627fc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100e20 0x7fef64196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:39548/0 (socket says 172.21.15.82:39548) 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef627fc640 1 -- 172.21.15.82:0/539060405 learned_addr learned my addr 172.21.15.82:0/539060405 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef62ffd640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef64100470 0x7fef64196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef627fc640 1 -- 172.21.15.82:0/539060405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef64100470 msgr2=0x7fef64196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef627fc640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef64100470 0x7fef64196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef627fc640 1 -- 172.21.15.82:0/539060405 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c009900 con 0x7fef64100e20 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.124+0000 7fef627fc640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100e20 0x7fef64196790 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00d6c0 tx=0x7fef4c00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:09.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.125+0000 7fef5bfff640 1 -- 172.21.15.82:0/539060405 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef4c0051f0 con 0x7fef64100e20 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.125+0000 7fef5bfff640 1 -- 172.21.15.82:0/539060405 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef4c009ce0 con 0x7fef64100e20 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.125+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6419add0 con 0x7fef64100e20 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.125+0000 7fef5bfff640 1 -- 172.21.15.82:0/539060405 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef4c010460 con 0x7fef64100e20 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.125+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6419b320 con 0x7fef64100e20 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.126+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef64101950 con 0x7fef64100e20 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.126+0000 7fef5bfff640 1 -- 172.21.15.82:0/539060405 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fef4c005350 con 0x7fef64100e20 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.126+0000 7fef5bfff640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fef5006aef0 0x7fef5006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.127+0000 7fef5bfff640 1 -- 172.21.15.82:0/539060405 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fef4c0896e0 con 0x7fef64100e20 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.127+0000 7fef62ffd640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fef5006aef0 0x7fef5006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:09.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.127+0000 7fef62ffd640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fef5006aef0 0x7fef5006d3b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fef5c005fa0 tx=0x7fef5c005f10 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:09.132 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.131+0000 7fef5bfff640 1 -- 172.21.15.82:0/539060405 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fef4c054cc0 con 0x7fef64100e20 2024-06-06T07:44:09.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.338+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fef6404e210 con 0x7fef64100e20 2024-06-06T07:44:09.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.338+0000 7fef5bfff640 1 -- 172.21.15.82:0/539060405 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fef4c0582e0 con 0x7fef64100e20 2024-06-06T07:44:09.339 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:09.339 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:09.340 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.340+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fef5006aef0 msgr2=0x7fef5006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.340+0000 7fef694c4640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fef5006aef0 0x7fef5006d3b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fef5c005fa0 tx=0x7fef5c005f10 comp rx=0 tx=0).stop 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.340+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100e20 msgr2=0x7fef64196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.340+0000 7fef694c4640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100e20 0x7fef64196790 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00d6c0 tx=0x7fef4c00db80 comp rx=0 tx=0).stop 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.340+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 shutdown_connections 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.340+0000 7fef694c4640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fef5006aef0 0x7fef5006d3b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.340+0000 7fef694c4640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef64100e20 0x7fef64196790 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.340+0000 7fef694c4640 1 --2- 172.21.15.82:0/539060405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef64100470 0x7fef64196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.340+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 >> 172.21.15.82:0/539060405 conn(0x7fef640fbd20 msgr2=0x7fef640fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.341+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 shutdown_connections 2024-06-06T07:44:09.341 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:09.341+0000 7fef694c4640 1 -- 172.21.15.82:0/539060405 wait complete. 2024-06-06T07:44:09.992 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:09 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/539060405' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:10.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:09 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/539060405' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:11.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:10 smithi082 ceph-mon[29354]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:11.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:10 smithi184 ceph-mon[37288]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:13.017 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:13.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:12 smithi082 ceph-mon[29354]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:13.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:12 smithi184 ceph-mon[37288]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:13.234 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:14.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.897+0000 7f67fddce640 1 -- 172.21.15.82:0/3593333282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 msgr2=0x7f67f8103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:14.902 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.897+0000 7f67fddce640 1 --2- 172.21.15.82:0/3593333282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 0x7f67f8103ff0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009880 tx=0x7f67ec02a530 comp rx=0 tx=0).stop 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.897+0000 7f67fddce640 1 -- 172.21.15.82:0/3593333282 shutdown_connections 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.897+0000 7f67fddce640 1 --2- 172.21.15.82:0/3593333282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 0x7f67f8103ff0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.897+0000 7f67fddce640 1 --2- 172.21.15.82:0/3593333282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67f80ff200 0x7f67f80ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.897+0000 7f67fddce640 1 -- 172.21.15.82:0/3593333282 >> 172.21.15.82:0/3593333282 conn(0x7f67f80fac80 msgr2=0x7f67f80fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.897+0000 7f67fddce640 1 -- 172.21.15.82:0/3593333282 shutdown_connections 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.897+0000 7f67fddce640 1 -- 172.21.15.82:0/3593333282 wait complete. 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67fddce640 1 Processor -- start 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67fddce640 1 -- start start 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67fddce640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67f80ff200 0x7f67f81a2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:14.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67fddce640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 0x7f67f81a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67fddce640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f819c7a0 con 0x7f67f80ffb20 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67fddce640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f819c910 con 0x7f67f80ff200 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67f6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 0x7f67f81a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67f6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 0x7f67f81a2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:39578/0 (socket says 172.21.15.82:39578) 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67f6ffd640 1 -- 172.21.15.82:0/1334176328 learned_addr learned my addr 172.21.15.82:0/1334176328 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67f6ffd640 1 -- 172.21.15.82:0/1334176328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67f80ff200 msgr2=0x7f67f81a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67f6ffd640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67f80ff200 0x7f67f81a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.898+0000 7f67f6ffd640 1 -- 172.21.15.82:0/1334176328 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec009560 con 0x7f67f80ffb20 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.899+0000 7f67f6ffd640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 0x7f67f81a2ba0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f67ec02aa50 tx=0x7f67ec0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.899+0000 7f67f4ff9640 1 -- 172.21.15.82:0/1334176328 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67ec004aa0 con 0x7f67f80ffb20 2024-06-06T07:44:14.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.899+0000 7f67f4ff9640 1 -- 172.21.15.82:0/1334176328 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67ec004c00 con 0x7f67f80ffb20 2024-06-06T07:44:14.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.899+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f819cb90 con 0x7f67f80ffb20 2024-06-06T07:44:14.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.899+0000 7f67f4ff9640 1 -- 172.21.15.82:0/1334176328 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67ec0039e0 con 0x7f67f80ffb20 2024-06-06T07:44:14.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.899+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f819d080 con 0x7f67f80ffb20 2024-06-06T07:44:14.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.900+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f8069060 con 0x7f67f80ffb20 2024-06-06T07:44:14.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.900+0000 7f67f4ff9640 1 -- 172.21.15.82:0/1334176328 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f67ec02ad50 con 0x7f67f80ffb20 2024-06-06T07:44:14.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.900+0000 7f67f4ff9640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f67e006aef0 0x7f67e006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:14.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.901+0000 7f67f4ff9640 1 -- 172.21.15.82:0/1334176328 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f67ec0aaa00 con 0x7f67f80ffb20 2024-06-06T07:44:14.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.901+0000 7f67f77fe640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f67e006aef0 0x7f67e006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:14.905 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.901+0000 7f67f77fe640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f67e006aef0 0x7f67e006d3b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f67e8009710 tx=0x7f67e8006c90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:14.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:14.905+0000 7f67f4ff9640 1 -- 172.21.15.82:0/1334176328 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f67ec075fe0 con 0x7f67f80ffb20 2024-06-06T07:44:15.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:14 smithi082 ceph-mon[29354]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:15.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:14 smithi184 ceph-mon[37288]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:15.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.118+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f67f819dd30 con 0x7f67f80ffb20 2024-06-06T07:44:15.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.118+0000 7f67f4ff9640 1 -- 172.21.15.82:0/1334176328 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f67ec079600 con 0x7f67f80ffb20 2024-06-06T07:44:15.118 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:15.118 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:15.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f67e006aef0 msgr2=0x7f67e006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:15.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f67e006aef0 0x7f67e006d3b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f67e8009710 tx=0x7f67e8006c90 comp rx=0 tx=0).stop 2024-06-06T07:44:15.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 msgr2=0x7f67f81a2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:15.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 0x7f67f81a2ba0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f67ec02aa50 tx=0x7f67ec0055a0 comp rx=0 tx=0).stop 2024-06-06T07:44:15.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 shutdown_connections 2024-06-06T07:44:15.121 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f67e006aef0 0x7f67e006d3b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:15.121 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67f80ffb20 0x7f67f81a2ba0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:15.121 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 --2- 172.21.15.82:0/1334176328 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67f80ff200 0x7f67f81a2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:15.121 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 >> 172.21.15.82:0/1334176328 conn(0x7f67f80fac80 msgr2=0x7f67f810ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:15.121 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 shutdown_connections 2024-06-06T07:44:15.121 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:15.120+0000 7f67fddce640 1 -- 172.21.15.82:0/1334176328 wait complete. 2024-06-06T07:44:16.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:15 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1334176328' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:16.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:15 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1334176328' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:17.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:16 smithi082 ceph-mon[29354]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:17.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:16 smithi184 ceph-mon[37288]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:18.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:19.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:18 smithi082 ceph-mon[29354]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:19.065 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:19.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:18 smithi184 ceph-mon[37288]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:20.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.677+0000 7f07b6f8b640 1 -- 172.21.15.82:0/2933632470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100e40 msgr2=0x7f07b0104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:20.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.677+0000 7f07b6f8b640 1 --2- 172.21.15.82:0/2933632470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100e40 0x7f07b0104e90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f079c009880 tx=0x7f079c02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:44:20.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.677+0000 7f07b6f8b640 1 -- 172.21.15.82:0/2933632470 shutdown_connections 2024-06-06T07:44:20.700 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.677+0000 7f07b6f8b640 1 --2- 172.21.15.82:0/2933632470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100e40 0x7f07b0104e90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:20.700 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.677+0000 7f07b6f8b640 1 --2- 172.21.15.82:0/2933632470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07b0100490 0x7f07b0100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:20.700 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b6f8b640 1 -- 172.21.15.82:0/2933632470 >> 172.21.15.82:0/2933632470 conn(0x7f07b00fbd40 msgr2=0x7f07b00fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:20.700 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b6f8b640 1 -- 172.21.15.82:0/2933632470 shutdown_connections 2024-06-06T07:44:20.700 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b6f8b640 1 -- 172.21.15.82:0/2933632470 wait complete. 2024-06-06T07:44:20.700 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b6f8b640 1 Processor -- start 2024-06-06T07:44:20.700 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b6f8b640 1 -- start start 2024-06-06T07:44:20.700 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b6f8b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100490 0x7f07b00727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:20.700 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b6f8b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07b0100e40 0x7f07b0072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b6f8b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0073230 con 0x7f07b0100490 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b6f8b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00733a0 con 0x7f07b0100e40 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b4d27640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100490 0x7f07b00727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b4d27640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100490 0x7f07b00727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:55636/0 (socket says 172.21.15.82:55636) 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.678+0000 7f07b4d27640 1 -- 172.21.15.82:0/397178191 learned_addr learned my addr 172.21.15.82:0/397178191 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.679+0000 7f07b4d27640 1 -- 172.21.15.82:0/397178191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07b0100e40 msgr2=0x7f07b0072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.679+0000 7f07b4d27640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07b0100e40 0x7f07b0072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.679+0000 7f07b4d27640 1 -- 172.21.15.82:0/397178191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c009560 con 0x7f07b0100490 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.679+0000 7f07b4d27640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100490 0x7f07b00727b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f07a400b420 tx=0x7f07a400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.679+0000 7f07adffb640 1 -- 172.21.15.82:0/397178191 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a4011800 con 0x7f07b0100490 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.679+0000 7f07adffb640 1 -- 172.21.15.82:0/397178191 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07a4011e20 con 0x7f07b0100490 2024-06-06T07:44:20.701 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.679+0000 7f07adffb640 1 -- 172.21.15.82:0/397178191 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a400f2e0 con 0x7f07b0100490 2024-06-06T07:44:20.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.679+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b01a2540 con 0x7f07b0100490 2024-06-06T07:44:20.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.679+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b01a28a0 con 0x7f07b0100490 2024-06-06T07:44:20.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.680+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b0101970 con 0x7f07b0100490 2024-06-06T07:44:20.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.681+0000 7f07adffb640 1 -- 172.21.15.82:0/397178191 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f07a4010380 con 0x7f07b0100490 2024-06-06T07:44:20.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.681+0000 7f07adffb640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f07a006aef0 0x7f07a006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:20.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.681+0000 7f07adffb640 1 -- 172.21.15.82:0/397178191 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f07a40894d0 con 0x7f07b0100490 2024-06-06T07:44:20.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.681+0000 7f07affff640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f07a006aef0 0x7f07a006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:20.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.681+0000 7f07affff640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f07a006aef0 0x7f07a006d3b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f07b0073ee0 tx=0x7f079c000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:20.702 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.685+0000 7f07adffb640 1 -- 172.21.15.82:0/397178191 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f07a4054b30 con 0x7f07b0100490 2024-06-06T07:44:20.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:20 smithi082 ceph-mon[29354]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:20.894 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.894+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f07b0073560 con 0x7f07b0100490 2024-06-06T07:44:20.894 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.894+0000 7f07adffb640 1 -- 172.21.15.82:0/397178191 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f07a4058150 con 0x7f07b0100490 2024-06-06T07:44:20.894 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:20.895 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:20.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f07a006aef0 msgr2=0x7f07a006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:20.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f07a006aef0 0x7f07a006d3b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f07b0073ee0 tx=0x7f079c000bc0 comp rx=0 tx=0).stop 2024-06-06T07:44:20.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100490 msgr2=0x7f07b00727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:20.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100490 0x7f07b00727b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f07a400b420 tx=0x7f07a400b8e0 comp rx=0 tx=0).stop 2024-06-06T07:44:20.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 shutdown_connections 2024-06-06T07:44:20.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f07a006aef0 0x7f07a006d3b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:20.896 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f07b0100e40 0x7f07b0072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:20.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 --2- 172.21.15.82:0/397178191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07b0100490 0x7f07b00727b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:20.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 >> 172.21.15.82:0/397178191 conn(0x7f07b00fbd40 msgr2=0x7f07b0103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:20.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 shutdown_connections 2024-06-06T07:44:20.897 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:20.896+0000 7f07b6f8b640 1 -- 172.21.15.82:0/397178191 wait complete. 2024-06-06T07:44:21.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:20 smithi184 ceph-mon[37288]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:22.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:21 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/397178191' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:22.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:21 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/397178191' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:22.843 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:22 smithi184 ceph-mon[37288]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:22.843 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:44:23.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:22 smithi082 ceph-mon[29354]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:23.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:44:24.584 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:24.802 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:24.924 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:24 smithi082 ceph-mon[29354]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:25.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:24 smithi184 ceph-mon[37288]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:26.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.458+0000 7fc7b9252640 1 -- 172.21.15.82:0/282672633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 msgr2=0x7fc7b4103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:26.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.458+0000 7fc7b9252640 1 --2- 172.21.15.82:0/282672633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 0x7fc7b4103d60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009880 tx=0x7fc7a802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:44:26.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.458+0000 7fc7b9252640 1 -- 172.21.15.82:0/282672633 shutdown_connections 2024-06-06T07:44:26.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.458+0000 7fc7b9252640 1 --2- 172.21.15.82:0/282672633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 0x7fc7b4103d60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:26.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.458+0000 7fc7b9252640 1 --2- 172.21.15.82:0/282672633 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7b40ff360 0x7fc7b40ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.458+0000 7fc7b9252640 1 -- 172.21.15.82:0/282672633 >> 172.21.15.82:0/282672633 conn(0x7fc7b4075e10 msgr2=0x7fc7b40fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.458+0000 7fc7b9252640 1 -- 172.21.15.82:0/282672633 shutdown_connections 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.458+0000 7fc7b9252640 1 -- 172.21.15.82:0/282672633 wait complete. 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.459+0000 7fc7b9252640 1 Processor -- start 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.459+0000 7fc7b9252640 1 -- start start 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b9252640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7b40ff360 0x7fc7b4196240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b9252640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 0x7fc7b4196780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b9252640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4196e10 con 0x7fc7b40ffd10 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b9252640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b419ab30 con 0x7fc7b40ff360 2024-06-06T07:44:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 0x7fc7b4196780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:26.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 0x7fc7b4196780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:39210/0 (socket says 172.21.15.82:39210) 2024-06-06T07:44:26.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b259c640 1 -- 172.21.15.82:0/237971682 learned_addr learned my addr 172.21.15.82:0/237971682 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:26.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b2d9d640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7b40ff360 0x7fc7b4196240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:26.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b259c640 1 -- 172.21.15.82:0/237971682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7b40ff360 msgr2=0x7fc7b4196240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:26.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b259c640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7b40ff360 0x7fc7b4196240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:26.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b259c640 1 -- 172.21.15.82:0/237971682 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a8009560 con 0x7fc7b40ffd10 2024-06-06T07:44:26.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b259c640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 0x7fc7b4196780 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc7a802a9c0 tx=0x7fc7a80055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:26.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc79bfff640 1 -- 172.21.15.82:0/237971682 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a8004aa0 con 0x7fc7b40ffd10 2024-06-06T07:44:26.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc79bfff640 1 -- 172.21.15.82:0/237971682 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7a8031470 con 0x7fc7b40ffd10 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b419adb0 con 0x7fc7b40ffd10 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc79bfff640 1 -- 172.21.15.82:0/237971682 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a80039e0 con 0x7fc7b40ffd10 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.460+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b419b2a0 con 0x7fc7b40ffd10 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.461+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b4100840 con 0x7fc7b40ffd10 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.462+0000 7fc79bfff640 1 -- 172.21.15.82:0/237971682 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fc7a8031c00 con 0x7fc7b40ffd10 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.462+0000 7fc79bfff640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc79c06aef0 0x7fc79c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.462+0000 7fc79bfff640 1 -- 172.21.15.82:0/237971682 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc7a80aac10 con 0x7fc7b40ffd10 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.462+0000 7fc7b2d9d640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc79c06aef0 0x7fc79c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.463+0000 7fc7b2d9d640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc79c06aef0 0x7fc79c06d3b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac005fd0 tx=0x7fc7ac00b390 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:26.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.467+0000 7fc79bfff640 1 -- 172.21.15.82:0/237971682 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc7a80762a0 con 0x7fc7b40ffd10 2024-06-06T07:44:26.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.678+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc7b404e210 con 0x7fc7b40ffd10 2024-06-06T07:44:26.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.678+0000 7fc79bfff640 1 -- 172.21.15.82:0/237971682 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc7a80798c0 con 0x7fc7b40ffd10 2024-06-06T07:44:26.678 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:26.679 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.680+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc79c06aef0 msgr2=0x7fc79c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.680+0000 7fc7b9252640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc79c06aef0 0x7fc79c06d3b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac005fd0 tx=0x7fc7ac00b390 comp rx=0 tx=0).stop 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.680+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 msgr2=0x7fc7b4196780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.680+0000 7fc7b9252640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 0x7fc7b4196780 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc7a802a9c0 tx=0x7fc7a80055a0 comp rx=0 tx=0).stop 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.680+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 shutdown_connections 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.680+0000 7fc7b9252640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fc79c06aef0 0x7fc79c06d3b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.681+0000 7fc7b9252640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7b40ffd10 0x7fc7b4196780 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.681+0000 7fc7b9252640 1 --2- 172.21.15.82:0/237971682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7b40ff360 0x7fc7b4196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.681+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 >> 172.21.15.82:0/237971682 conn(0x7fc7b4075e10 msgr2=0x7fc7b40fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.681+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 shutdown_connections 2024-06-06T07:44:26.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:26.681+0000 7fc7b9252640 1 -- 172.21.15.82:0/237971682 wait complete. 2024-06-06T07:44:26.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:26 smithi082 ceph-mon[29354]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:26.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:26 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/237971682' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:27.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:26 smithi184 ceph-mon[37288]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:27.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:26 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/237971682' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:29.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:28 smithi082 ceph-mon[29354]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:29.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:28 smithi184 ceph-mon[37288]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:30.424 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:30.642 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:31.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:30 smithi082 ceph-mon[29354]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:31.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:30 smithi184 ceph-mon[37288]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:32.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.224+0000 7f9a7c457640 1 -- 172.21.15.82:0/911602405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 msgr2=0x7f9a7410a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:32.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.224+0000 7f9a7c457640 1 --2- 172.21.15.82:0/911602405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 0x7f9a7410a270 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009880 tx=0x7f9a7002a530 comp rx=0 tx=0).stop 2024-06-06T07:44:32.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.224+0000 7f9a7c457640 1 -- 172.21.15.82:0/911602405 shutdown_connections 2024-06-06T07:44:32.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.224+0000 7f9a7c457640 1 --2- 172.21.15.82:0/911602405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 0x7f9a7410a270 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.224+0000 7f9a7c457640 1 --2- 172.21.15.82:0/911602405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a74101460 0x7f9a74101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.224+0000 7f9a7c457640 1 -- 172.21.15.82:0/911602405 >> 172.21.15.82:0/911602405 conn(0x7f9a740fac50 msgr2=0x7f9a740fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.224+0000 7f9a7c457640 1 -- 172.21.15.82:0/911602405 shutdown_connections 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.224+0000 7f9a7c457640 1 -- 172.21.15.82:0/911602405 wait complete. 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a7c457640 1 Processor -- start 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a7c457640 1 -- start start 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a7c457640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a74101460 0x7f9a740ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a7c457640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 0x7f9a740ff600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a7c457640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740ffb40 con 0x7f9a74101d80 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a7c457640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740ffcb0 con 0x7f9a74101460 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a799f2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 0x7f9a740ff600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a799f2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 0x7f9a740ff600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:39222/0 (socket says 172.21.15.82:39222) 2024-06-06T07:44:32.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a799f2640 1 -- 172.21.15.82:0/2156428681 learned_addr learned my addr 172.21.15.82:0/2156428681 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a799f2640 1 -- 172.21.15.82:0/2156428681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a74101460 msgr2=0x7f9a740ff0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a799f2640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a74101460 0x7f9a740ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a799f2640 1 -- 172.21.15.82:0/2156428681 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a70009560 con 0x7f9a74101d80 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.225+0000 7f9a799f2640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 0x7f9a740ff600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006010 tx=0x7f9a700055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.226+0000 7f9a6b7fe640 1 -- 172.21.15.82:0/2156428681 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a70004aa0 con 0x7f9a74101d80 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.226+0000 7f9a6b7fe640 1 -- 172.21.15.82:0/2156428681 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a70004c00 con 0x7f9a74101d80 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.226+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a740fff30 con 0x7f9a74101d80 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.226+0000 7f9a6b7fe640 1 -- 172.21.15.82:0/2156428681 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a70003a20 con 0x7f9a74101d80 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.226+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a741a2760 con 0x7f9a74101d80 2024-06-06T07:44:32.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.227+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a74069020 con 0x7f9a74101d80 2024-06-06T07:44:32.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.227+0000 7f9a6b7fe640 1 -- 172.21.15.82:0/2156428681 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f9a7002ad50 con 0x7f9a74101d80 2024-06-06T07:44:32.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.227+0000 7f9a6b7fe640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9a6006af00 0x7f9a6006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:32.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.228+0000 7f9a7a1f3640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9a6006af00 0x7f9a6006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:32.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.228+0000 7f9a6b7fe640 1 -- 172.21.15.82:0/2156428681 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9a700aa9e0 con 0x7f9a74101d80 2024-06-06T07:44:32.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.228+0000 7f9a7a1f3640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9a6006af00 0x7f9a6006d3c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009710 tx=0x7f9a64006c90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:32.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.232+0000 7f9a6b7fe640 1 -- 172.21.15.82:0/2156428681 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9a700760e0 con 0x7f9a74101d80 2024-06-06T07:44:32.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.442+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9a740fff30 con 0x7f9a74101d80 2024-06-06T07:44:32.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.443+0000 7f9a6b7fe640 1 -- 172.21.15.82:0/2156428681 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9a7004a0d0 con 0x7f9a74101d80 2024-06-06T07:44:32.443 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:32.443 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:32.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9a6006af00 msgr2=0x7f9a6006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:32.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9a6006af00 0x7f9a6006d3c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009710 tx=0x7f9a64006c90 comp rx=0 tx=0).stop 2024-06-06T07:44:32.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 msgr2=0x7f9a740ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:32.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 0x7f9a740ff600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006010 tx=0x7f9a700055a0 comp rx=0 tx=0).stop 2024-06-06T07:44:32.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 shutdown_connections 2024-06-06T07:44:32.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9a6006af00 0x7f9a6006d3c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:32.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a74101d80 0x7f9a740ff600 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:32.445 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 --2- 172.21.15.82:0/2156428681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a74101460 0x7f9a740ff0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:32.446 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 >> 172.21.15.82:0/2156428681 conn(0x7f9a740fac50 msgr2=0x7f9a741026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:32.446 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 shutdown_connections 2024-06-06T07:44:32.446 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:32.445+0000 7f9a7c457640 1 -- 172.21.15.82:0/2156428681 wait complete. 2024-06-06T07:44:33.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:32 smithi082 ceph-mon[29354]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:33.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:32 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2156428681' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:33.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:32 smithi184 ceph-mon[37288]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:33.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:32 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2156428681' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:35.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:34 smithi082 ceph-mon[29354]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:35.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:34 smithi184 ceph-mon[37288]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:36.211 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:36.428 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:36.859 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:36 smithi082 ceph-mon[29354]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:37.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:36 smithi184 ceph-mon[37288]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:37.874 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:44:38.083 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.079+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3376204979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 msgr2=0x7fcb24104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:38.083 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.079+0000 7fcb2c5c1640 1 --2- 172.21.15.82:0/3376204979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 0x7fcb24104eb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009880 tx=0x7fcb1402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.079+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3376204979 shutdown_connections 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.079+0000 7fcb2c5c1640 1 --2- 172.21.15.82:0/3376204979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 0x7fcb24104eb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.079+0000 7fcb2c5c1640 1 --2- 172.21.15.82:0/3376204979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb241004b0 0x7fcb24100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.079+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3376204979 >> 172.21.15.82:0/3376204979 conn(0x7fcb240fbd40 msgr2=0x7fcb240fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.079+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3376204979 shutdown_connections 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.079+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3376204979 wait complete. 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.080+0000 7fcb2c5c1640 1 Processor -- start 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.080+0000 7fcb2c5c1640 1 -- start start 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.080+0000 7fcb2c5c1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb241004b0 0x7fcb24196230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.080+0000 7fcb2c5c1640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 0x7fcb24196770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.080+0000 7fcb2c5c1640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb24196e00 con 0x7fcb24100e60 2024-06-06T07:44:38.084 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.080+0000 7fcb2c5c1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2419aad0 con 0x7fcb241004b0 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.080+0000 7fcb29b5c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 0x7fcb24196770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb29b5c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 0x7fcb24196770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:35450/0 (socket says 172.21.15.82:35450) 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb29b5c640 1 -- 172.21.15.82:0/3526408339 learned_addr learned my addr 172.21.15.82:0/3526408339 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb2a35d640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb241004b0 0x7fcb24196230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb29b5c640 1 -- 172.21.15.82:0/3526408339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb241004b0 msgr2=0x7fcb24196230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb29b5c640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb241004b0 0x7fcb24196230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb29b5c640 1 -- 172.21.15.82:0/3526408339 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb14009560 con 0x7fcb24100e60 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb29b5c640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 0x7fcb24196770 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcb1402a9c0 tx=0x7fcb140055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb1b7fe640 1 -- 172.21.15.82:0/3526408339 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb14004aa0 con 0x7fcb24100e60 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb1b7fe640 1 -- 172.21.15.82:0/3526408339 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb14004c00 con 0x7fcb24100e60 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2419ad50 con 0x7fcb24100e60 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb1b7fe640 1 -- 172.21.15.82:0/3526408339 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb14003a50 con 0x7fcb24100e60 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.081+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb2419b240 con 0x7fcb24100e60 2024-06-06T07:44:38.085 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.082+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb24101990 con 0x7fcb24100e60 2024-06-06T07:44:38.086 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.083+0000 7fcb1b7fe640 1 -- 172.21.15.82:0/3526408339 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fcb14031c90 con 0x7fcb24100e60 2024-06-06T07:44:38.086 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.084+0000 7fcb1b7fe640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcb1006aef0 0x7fcb1006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:38.086 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.084+0000 7fcb1b7fe640 1 -- 172.21.15.82:0/3526408339 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fcb140aad10 con 0x7fcb24100e60 2024-06-06T07:44:38.086 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.084+0000 7fcb2a35d640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcb1006aef0 0x7fcb1006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:38.086 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.084+0000 7fcb2a35d640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcb1006aef0 0x7fcb1006d3b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcb200058c0 tx=0x7fcb2000a2d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:38.088 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.088+0000 7fcb1b7fe640 1 -- 172.21.15.82:0/3526408339 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fcb140763a0 con 0x7fcb24100e60 2024-06-06T07:44:38.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:44:38.326 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.326+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcb2404e210 con 0x7fcb24100e60 2024-06-06T07:44:38.327 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.326+0000 7fcb1b7fe640 1 -- 172.21.15.82:0/3526408339 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcb140799c0 con 0x7fcb24100e60 2024-06-06T07:44:38.327 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:38.327 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.328+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcb1006aef0 msgr2=0x7fcb1006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.328+0000 7fcb2c5c1640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcb1006aef0 0x7fcb1006d3b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcb200058c0 tx=0x7fcb2000a2d0 comp rx=0 tx=0).stop 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.328+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 msgr2=0x7fcb24196770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.328+0000 7fcb2c5c1640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 0x7fcb24196770 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcb1402a9c0 tx=0x7fcb140055a0 comp rx=0 tx=0).stop 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.329+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 shutdown_connections 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.329+0000 7fcb2c5c1640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fcb1006aef0 0x7fcb1006d3b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.329+0000 7fcb2c5c1640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb24100e60 0x7fcb24196770 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.329+0000 7fcb2c5c1640 1 --2- 172.21.15.82:0/3526408339 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcb241004b0 0x7fcb24196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.329+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 >> 172.21.15.82:0/3526408339 conn(0x7fcb240fbd40 msgr2=0x7fcb240fd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:38.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.329+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 shutdown_connections 2024-06-06T07:44:38.330 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:38.329+0000 7fcb2c5c1640 1 -- 172.21.15.82:0/3526408339 wait complete. 2024-06-06T07:44:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:38 smithi082 ceph-mon[29354]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:38.944 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:38 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3526408339' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:39.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:38 smithi184 ceph-mon[37288]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:39.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:38 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3526408339' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:41.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:40 smithi082 ceph-mon[29354]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:41.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:40 smithi184 ceph-mon[37288]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:41.965 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:42.182 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:43.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:42 smithi082 ceph-mon[29354]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:43.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:42 smithi184 ceph-mon[37288]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:43.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.846+0000 7f2094588640 1 -- 172.21.15.82:0/3437624820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 msgr2=0x7f208c100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:43.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.846+0000 7f2094588640 1 --2- 172.21.15.82:0/3437624820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 0x7f208c100850 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2088009880 tx=0x7f208802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:44:43.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.846+0000 7f2094588640 1 -- 172.21.15.82:0/3437624820 shutdown_connections 2024-06-06T07:44:43.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.846+0000 7f2094588640 1 --2- 172.21.15.82:0/3437624820 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f208c100e20 0x7f208c104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:43.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.846+0000 7f2094588640 1 --2- 172.21.15.82:0/3437624820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 0x7f208c100850 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:43.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.846+0000 7f2094588640 1 -- 172.21.15.82:0/3437624820 >> 172.21.15.82:0/3437624820 conn(0x7f208c0fbd20 msgr2=0x7f208c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:43.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.846+0000 7f2094588640 1 -- 172.21.15.82:0/3437624820 shutdown_connections 2024-06-06T07:44:43.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.846+0000 7f2094588640 1 -- 172.21.15.82:0/3437624820 wait complete. 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.847+0000 7f2094588640 1 Processor -- start 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.847+0000 7f2094588640 1 -- start start 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.847+0000 7f2094588640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 0x7f208c196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.847+0000 7f2094588640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f208c100e20 0x7f208c1967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.847+0000 7f2094588640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c196db0 con 0x7f208c100470 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.847+0000 7f2094588640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c196f20 con 0x7f208c100e20 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.847+0000 7f2092324640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 0x7f208c196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.847+0000 7f2092324640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 0x7f208c196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:35472/0 (socket says 172.21.15.82:35472) 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.847+0000 7f2092324640 1 -- 172.21.15.82:0/3033333214 learned_addr learned my addr 172.21.15.82:0/3033333214 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.848+0000 7f2092324640 1 -- 172.21.15.82:0/3033333214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f208c100e20 msgr2=0x7f208c1967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.848+0000 7f2092324640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f208c100e20 0x7f208c1967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:43.855 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.848+0000 7f2092324640 1 -- 172.21.15.82:0/3033333214 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2088009560 con 0x7f208c100470 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.848+0000 7f2092324640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 0x7f208c196270 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f208802a9c0 tx=0x7f20880055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.848+0000 7f20837fe640 1 -- 172.21.15.82:0/3033333214 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2088004aa0 con 0x7f208c100470 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.848+0000 7f20837fe640 1 -- 172.21.15.82:0/3033333214 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2088004c00 con 0x7f208c100470 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.848+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c19ac00 con 0x7f208c100470 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.848+0000 7f20837fe640 1 -- 172.21.15.82:0/3033333214 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2088003ab0 con 0x7f208c100470 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.848+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c19b0f0 con 0x7f208c100470 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.849+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c101950 con 0x7f208c100470 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.850+0000 7f20837fe640 1 -- 172.21.15.82:0/3033333214 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f2088031d20 con 0x7f208c100470 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.850+0000 7f20837fe640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f207806aef0 0x7f207806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.850+0000 7f20837fe640 1 -- 172.21.15.82:0/3033333214 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f20880aad10 con 0x7f208c100470 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.850+0000 7f2091b23640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f207806aef0 0x7f207806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:43.856 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.850+0000 7f2091b23640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f207806aef0 0x7f207806d3b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f208c197780 tx=0x7f207c005c70 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:43.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:43.854+0000 7f20837fe640 1 -- 172.21.15.82:0/3033333214 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f20880762f0 con 0x7f208c100470 2024-06-06T07:44:44.062 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.061+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f208c04e210 con 0x7f208c100470 2024-06-06T07:44:44.062 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.061+0000 7f20837fe640 1 -- 172.21.15.82:0/3033333214 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2088079910 con 0x7f208c100470 2024-06-06T07:44:44.063 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:44.063 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.063+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f207806aef0 msgr2=0x7f207806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.063+0000 7f2094588640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f207806aef0 0x7f207806d3b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f208c197780 tx=0x7f207c005c70 comp rx=0 tx=0).stop 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.063+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 msgr2=0x7f208c196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.063+0000 7f2094588640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 0x7f208c196270 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f208802a9c0 tx=0x7f20880055a0 comp rx=0 tx=0).stop 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.063+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 shutdown_connections 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.063+0000 7f2094588640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f207806aef0 0x7f207806d3b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.063+0000 7f2094588640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f208c100e20 0x7f208c1967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.063+0000 7f2094588640 1 --2- 172.21.15.82:0/3033333214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f208c100470 0x7f208c196270 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.064+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 >> 172.21.15.82:0/3033333214 conn(0x7f208c0fbd20 msgr2=0x7f208c0fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.064+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 shutdown_connections 2024-06-06T07:44:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:44.064+0000 7f2094588640 1 -- 172.21.15.82:0/3033333214 wait complete. 2024-06-06T07:44:44.932 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:44 smithi082 ceph-mon[29354]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:44.932 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:44 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3033333214' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:45.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:44 smithi184 ceph-mon[37288]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:45.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:44 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3033333214' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:47.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:46 smithi082 ceph-mon[29354]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:47.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:46 smithi184 ceph-mon[37288]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:47.956 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:48.163 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:49.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:48 smithi082 ceph-mon[29354]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:49.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:48 smithi184 ceph-mon[37288]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:49.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.862+0000 7f25ae197640 1 -- 172.21.15.82:0/864633658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 msgr2=0x7f25a80ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:49.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.862+0000 7f25ae197640 1 --2- 172.21.15.82:0/864633658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 0x7f25a80ff790 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2598009880 tx=0x7f259802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:44:49.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.862+0000 7f25ae197640 1 -- 172.21.15.82:0/864633658 shutdown_connections 2024-06-06T07:44:49.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.862+0000 7f25ae197640 1 --2- 172.21.15.82:0/864633658 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25a80ffd60 0x7f25a8103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.862+0000 7f25ae197640 1 --2- 172.21.15.82:0/864633658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 0x7f25a80ff790 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.862+0000 7f25ae197640 1 -- 172.21.15.82:0/864633658 >> 172.21.15.82:0/864633658 conn(0x7f25a80fac60 msgr2=0x7f25a80fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.862+0000 7f25ae197640 1 -- 172.21.15.82:0/864633658 shutdown_connections 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.862+0000 7f25ae197640 1 -- 172.21.15.82:0/864633658 wait complete. 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.863+0000 7f25ae197640 1 Processor -- start 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.863+0000 7f25ae197640 1 -- start start 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.863+0000 7f25ae197640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 0x7f25a81970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.863+0000 7f25ae197640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25a80ffd60 0x7f25a8197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.863+0000 7f25ae197640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a8197b60 con 0x7f25a80ff3b0 2024-06-06T07:44:49.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.863+0000 7f25ae197640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a8197cd0 con 0x7f25a80ffd60 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.863+0000 7f25a77fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 0x7f25a81970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.863+0000 7f25a77fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 0x7f25a81970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:38120/0 (socket says 172.21.15.82:38120) 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.863+0000 7f25a77fe640 1 -- 172.21.15.82:0/858306991 learned_addr learned my addr 172.21.15.82:0/858306991 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f259effd640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25a80ffd60 0x7f25a8197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f25a77fe640 1 -- 172.21.15.82:0/858306991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25a80ffd60 msgr2=0x7f25a8197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f25a77fe640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25a80ffd60 0x7f25a8197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f25a77fe640 1 -- 172.21.15.82:0/858306991 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598009560 con 0x7f25a80ff3b0 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f259effd640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25a80ffd60 0x7f25a8197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f25a77fe640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 0x7f25a81970e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f259802a9c0 tx=0x7f25980055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:49.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f25a57fa640 1 -- 172.21.15.82:0/858306991 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2598004aa0 con 0x7f25a80ff3b0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a8197f50 con 0x7f25a80ff3b0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f25a57fa640 1 -- 172.21.15.82:0/858306991 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2598004c00 con 0x7f25a80ff3b0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f25a57fa640 1 -- 172.21.15.82:0/858306991 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2598003ab0 con 0x7f25a80ff3b0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.864+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a81a0430 con 0x7f25a80ff3b0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.865+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a8100890 con 0x7f25a80ff3b0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.866+0000 7f25a57fa640 1 -- 172.21.15.82:0/858306991 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f2598003ce0 con 0x7f25a80ff3b0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.866+0000 7f25a57fa640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f259406aef0 0x7f259406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.866+0000 7f25a57fa640 1 -- 172.21.15.82:0/858306991 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f25980aabd0 con 0x7f25a80ff3b0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.866+0000 7f259effd640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f259406aef0 0x7f259406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.866+0000 7f259effd640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f259406aef0 0x7f259406d3b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f25a8068f20 tx=0x7f2590005d30 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:49.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:49.870+0000 7f25a57fa640 1 -- 172.21.15.82:0/858306991 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f25980761b0 con 0x7f25a80ff3b0 2024-06-06T07:44:50.078 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.078+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f25a8068cd0 con 0x7f25a80ff3b0 2024-06-06T07:44:50.079 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.079+0000 7f25a57fa640 1 -- 172.21.15.82:0/858306991 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f25980797d0 con 0x7f25a80ff3b0 2024-06-06T07:44:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:50.079 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f259406aef0 msgr2=0x7f259406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f259406aef0 0x7f259406d3b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f25a8068f20 tx=0x7f2590005d30 comp rx=0 tx=0).stop 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 msgr2=0x7f25a81970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 0x7f25a81970e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f259802a9c0 tx=0x7f25980055a0 comp rx=0 tx=0).stop 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 shutdown_connections 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f259406aef0 0x7f259406d3b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25a80ffd60 0x7f25a8197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 --2- 172.21.15.82:0/858306991 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25a80ff3b0 0x7f25a81970e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 >> 172.21.15.82:0/858306991 conn(0x7f25a80fac60 msgr2=0x7f25a80fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 shutdown_connections 2024-06-06T07:44:50.081 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:50.081+0000 7f25ae197640 1 -- 172.21.15.82:0/858306991 wait complete. 2024-06-06T07:44:51.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:50 smithi082 ceph-mon[29354]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:51.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:50 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/858306991' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:51.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:50 smithi184 ceph-mon[37288]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:51.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:50 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/858306991' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:53.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:52 smithi184 ceph-mon[37288]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:44:53.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:52 smithi082 ceph-mon[29354]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:53.269 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:44:53.775 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:53.989 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:44:55.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:54 smithi184 ceph-mon[37288]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:55.158 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:54 smithi082 ceph-mon[29354]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:55.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.641+0000 7fddcb907640 1 -- 172.21.15.82:0/579697562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101d20 msgr2=0x7fddc410a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:55.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.641+0000 7fddcb907640 1 --2- 172.21.15.82:0/579697562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101d20 0x7fddc410a210 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fddc0009880 tx=0x7fddc002a530 comp rx=0 tx=0).stop 2024-06-06T07:44:55.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.641+0000 7fddcb907640 1 -- 172.21.15.82:0/579697562 shutdown_connections 2024-06-06T07:44:55.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.641+0000 7fddcb907640 1 --2- 172.21.15.82:0/579697562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101d20 0x7fddc410a210 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:55.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.641+0000 7fddcb907640 1 --2- 172.21.15.82:0/579697562 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4101400 0x7fddc41017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:55.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.641+0000 7fddcb907640 1 -- 172.21.15.82:0/579697562 >> 172.21.15.82:0/579697562 conn(0x7fddc40fac50 msgr2=0x7fddc40fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:55.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.641+0000 7fddcb907640 1 -- 172.21.15.82:0/579697562 shutdown_connections 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.641+0000 7fddcb907640 1 -- 172.21.15.82:0/579697562 wait complete. 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.641+0000 7fddcb907640 1 Processor -- start 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddcb907640 1 -- start start 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddcb907640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101400 0x7fddc4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddcb907640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4101d20 0x7fddc4198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddcb907640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc4199160 con 0x7fddc4101400 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddcb907640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc419ce80 con 0x7fddc4101d20 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddc96a3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101400 0x7fddc4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddc96a3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101400 0x7fddc4198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:51418/0 (socket says 172.21.15.82:51418) 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddc96a3640 1 -- 172.21.15.82:0/3578005519 learned_addr learned my addr 172.21.15.82:0/3578005519 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddc96a3640 1 -- 172.21.15.82:0/3578005519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4101d20 msgr2=0x7fddc4198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:55.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddc8ea2640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4101d20 0x7fddc4198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddc96a3640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4101d20 0x7fddc4198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddc96a3640 1 -- 172.21.15.82:0/3578005519 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc0009560 con 0x7fddc4101400 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddc96a3640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101400 0x7fddc4198590 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fddb400b420 tx=0x7fddb400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.642+0000 7fddba7fc640 1 -- 172.21.15.82:0/3578005519 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddb4011800 con 0x7fddc4101400 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.643+0000 7fddba7fc640 1 -- 172.21.15.82:0/3578005519 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddb4011e20 con 0x7fddc4101400 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.643+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc419d160 con 0x7fddc4101400 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.643+0000 7fddba7fc640 1 -- 172.21.15.82:0/3578005519 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddb400f2e0 con 0x7fddc4101400 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.643+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc419d6b0 con 0x7fddc4101400 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.643+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc40690a0 con 0x7fddc4101400 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.644+0000 7fddba7fc640 1 -- 172.21.15.82:0/3578005519 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fddb4010380 con 0x7fddc4101400 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.644+0000 7fddba7fc640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fddb006aef0 0x7fddb006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.644+0000 7fddc8ea2640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fddb006aef0 0x7fddb006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.644+0000 7fddba7fc640 1 -- 172.21.15.82:0/3578005519 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fddb4089570 con 0x7fddc4101400 2024-06-06T07:44:55.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.645+0000 7fddc8ea2640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fddb006aef0 0x7fddb006d3b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fddc0000c00 tx=0x7fddc0004120 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:44:55.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.649+0000 7fddba7fc640 1 -- 172.21.15.82:0/3578005519 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fddb4054b50 con 0x7fddc4101400 2024-06-06T07:44:55.860 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:44:55.861 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:44:55.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.860+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fddc404e210 con 0x7fddc4101400 2024-06-06T07:44:55.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.860+0000 7fddba7fc640 1 -- 172.21.15.82:0/3578005519 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fddb4058170 con 0x7fddc4101400 2024-06-06T07:44:55.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fddb006aef0 msgr2=0x7fddb006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:55.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fddb006aef0 0x7fddb006d3b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fddc0000c00 tx=0x7fddc0004120 comp rx=0 tx=0).stop 2024-06-06T07:44:55.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101400 msgr2=0x7fddc4198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:44:55.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101400 0x7fddc4198590 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fddb400b420 tx=0x7fddb400b8e0 comp rx=0 tx=0).stop 2024-06-06T07:44:55.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 shutdown_connections 2024-06-06T07:44:55.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fddb006aef0 0x7fddb006d3b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:55.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fddc4101d20 0x7fddc4198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:55.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 --2- 172.21.15.82:0/3578005519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fddc4101400 0x7fddc4198590 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:44:55.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 >> 172.21.15.82:0/3578005519 conn(0x7fddc40fac50 msgr2=0x7fddc41006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:44:55.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 shutdown_connections 2024-06-06T07:44:55.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:44:55.862+0000 7fddcb907640 1 -- 172.21.15.82:0/3578005519 wait complete. 2024-06-06T07:44:57.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:56 smithi184 ceph-mon[37288]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:57.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:56 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3578005519' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:57.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:56 smithi082 ceph-mon[29354]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:57.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:56 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3578005519' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:44:59.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:44:58 smithi184 ceph-mon[37288]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:59.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:44:58 smithi082 ceph-mon[29354]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:44:59.555 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:44:59.772 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:01.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:00 smithi184 ceph-mon[37288]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:01.110 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:00 smithi082 ceph-mon[29354]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:01.307 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.304+0000 7fd1fec11640 1 -- 172.21.15.82:0/1614094861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 msgr2=0x7fd1f8103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.304+0000 7fd1fec11640 1 --2- 172.21.15.82:0/1614094861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 0x7fd1f8103db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009880 tx=0x7fd1e002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.304+0000 7fd1fec11640 1 -- 172.21.15.82:0/1614094861 shutdown_connections 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.304+0000 7fd1fec11640 1 --2- 172.21.15.82:0/1614094861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 0x7fd1f8103db0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.304+0000 7fd1fec11640 1 --2- 172.21.15.82:0/1614094861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80ff3b0 0x7fd1f80ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.304+0000 7fd1fec11640 1 -- 172.21.15.82:0/1614094861 >> 172.21.15.82:0/1614094861 conn(0x7fd1f80fac80 msgr2=0x7fd1f80fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.304+0000 7fd1fec11640 1 -- 172.21.15.82:0/1614094861 shutdown_connections 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.304+0000 7fd1fec11640 1 -- 172.21.15.82:0/1614094861 wait complete. 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1fec11640 1 Processor -- start 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1fec11640 1 -- start start 2024-06-06T07:45:01.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1fec11640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80ff3b0 0x7fd1f8197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1fec11640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 0x7fd1f81975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1fec11640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f8197b00 con 0x7fd1f80ffd60 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1fec11640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f8197c70 con 0x7fd1f80ff3b0 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1f7fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 0x7fd1f81975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1f7fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 0x7fd1f81975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:51430/0 (socket says 172.21.15.82:51430) 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1f7fff640 1 -- 172.21.15.82:0/4269287106 learned_addr learned my addr 172.21.15.82:0/4269287106 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1fc9ad640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80ff3b0 0x7fd1f8197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1f7fff640 1 -- 172.21.15.82:0/4269287106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80ff3b0 msgr2=0x7fd1f8197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1f7fff640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80ff3b0 0x7fd1f8197080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.305+0000 7fd1f7fff640 1 -- 172.21.15.82:0/4269287106 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e0009560 con 0x7fd1f80ffd60 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.306+0000 7fd1f7fff640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 0x7fd1f81975c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd1e002a9c0 tx=0x7fd1e00055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.306+0000 7fd1f5ffb640 1 -- 172.21.15.82:0/4269287106 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e0004aa0 con 0x7fd1f80ffd60 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.306+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f8197ef0 con 0x7fd1f80ffd60 2024-06-06T07:45:01.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.306+0000 7fd1f5ffb640 1 -- 172.21.15.82:0/4269287106 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1e0031470 con 0x7fd1f80ffd60 2024-06-06T07:45:01.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.306+0000 7fd1f5ffb640 1 -- 172.21.15.82:0/4269287106 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e00039e0 con 0x7fd1f80ffd60 2024-06-06T07:45:01.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.306+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f81a0420 con 0x7fd1f80ffd60 2024-06-06T07:45:01.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.307+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f8100890 con 0x7fd1f80ffd60 2024-06-06T07:45:01.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.307+0000 7fd1f5ffb640 1 -- 172.21.15.82:0/4269287106 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fd1e0031c00 con 0x7fd1f80ffd60 2024-06-06T07:45:01.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.308+0000 7fd1f5ffb640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd1e406aef0 0x7fd1e406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:01.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.308+0000 7fd1fc9ad640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd1e406aef0 0x7fd1e406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:01.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.308+0000 7fd1f5ffb640 1 -- 172.21.15.82:0/4269287106 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd1e00aab50 con 0x7fd1f80ffd60 2024-06-06T07:45:01.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.308+0000 7fd1fc9ad640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd1e406aef0 0x7fd1e406d3b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8005fd0 tx=0x7fd1e8005850 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:01.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.312+0000 7fd1f5ffb640 1 -- 172.21.15.82:0/4269287106 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd1e0076130 con 0x7fd1f80ffd60 2024-06-06T07:45:01.519 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.519+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd1f804e210 con 0x7fd1f80ffd60 2024-06-06T07:45:01.520 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.520+0000 7fd1f5ffb640 1 -- 172.21.15.82:0/4269287106 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd1e0079750 con 0x7fd1f80ffd60 2024-06-06T07:45:01.520 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:01.520 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.522+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd1e406aef0 msgr2=0x7fd1e406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.522+0000 7fd1fec11640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd1e406aef0 0x7fd1e406d3b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8005fd0 tx=0x7fd1e8005850 comp rx=0 tx=0).stop 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.522+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 msgr2=0x7fd1f81975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.522+0000 7fd1fec11640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 0x7fd1f81975c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd1e002a9c0 tx=0x7fd1e00055a0 comp rx=0 tx=0).stop 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.522+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 shutdown_connections 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.522+0000 7fd1fec11640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd1e406aef0 0x7fd1e406d3b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.522+0000 7fd1fec11640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1f80ffd60 0x7fd1f81975c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.522+0000 7fd1fec11640 1 --2- 172.21.15.82:0/4269287106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1f80ff3b0 0x7fd1f8197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.523+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 >> 172.21.15.82:0/4269287106 conn(0x7fd1f80fac80 msgr2=0x7fd1f80fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.523+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 shutdown_connections 2024-06-06T07:45:01.523 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:01.523+0000 7fd1fec11640 1 -- 172.21.15.82:0/4269287106 wait complete. 2024-06-06T07:45:02.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:01 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/4269287106' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:02.194 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:01 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/4269287106' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:03.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:02 smithi184 ceph-mon[37288]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:03.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:02 smithi082 ceph-mon[29354]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:05.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:04 smithi184 ceph-mon[37288]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:05.217 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:05.246 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:04 smithi082 ceph-mon[29354]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:05.435 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:06.896 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:06 smithi082 ceph-mon[29354]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:06.896 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:06 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:45:07.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.091+0000 7f0939ae5640 1 -- 172.21.15.82:0/575365615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 msgr2=0x7f0934103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:07.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.091+0000 7f0939ae5640 1 --2- 172.21.15.82:0/575365615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 0x7f0934103db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0928009880 tx=0x7f092802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:45:07.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.092+0000 7f0939ae5640 1 -- 172.21.15.82:0/575365615 shutdown_connections 2024-06-06T07:45:07.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.092+0000 7f0939ae5640 1 --2- 172.21.15.82:0/575365615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 0x7f0934103db0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:07.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.092+0000 7f0939ae5640 1 --2- 172.21.15.82:0/575365615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09340ff3b0 0x7f09340ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:07.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.092+0000 7f0939ae5640 1 -- 172.21.15.82:0/575365615 >> 172.21.15.82:0/575365615 conn(0x7f09340fac80 msgr2=0x7f09340fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:07.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.092+0000 7f0939ae5640 1 -- 172.21.15.82:0/575365615 shutdown_connections 2024-06-06T07:45:07.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.092+0000 7f0939ae5640 1 -- 172.21.15.82:0/575365615 wait complete. 2024-06-06T07:45:07.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.092+0000 7f0939ae5640 1 Processor -- start 2024-06-06T07:45:07.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.092+0000 7f0939ae5640 1 -- start start 2024-06-06T07:45:07.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:06 smithi184 ceph-mon[37288]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:07.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:06 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:45:07.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0939ae5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09340ff3b0 0x7f0934195290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:07.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0939ae5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 0x7f09341957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:07.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0939ae5640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0934195e60 con 0x7f09340ffd60 2024-06-06T07:45:07.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0939ae5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0934199b80 con 0x7f09340ff3b0 2024-06-06T07:45:07.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0932ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 0x7f09341957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:07.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0932ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 0x7f09341957d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:54910/0 (socket says 172.21.15.82:54910) 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0932ffd640 1 -- 172.21.15.82:0/2938587666 learned_addr learned my addr 172.21.15.82:0/2938587666 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f09337fe640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09340ff3b0 0x7f0934195290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0932ffd640 1 -- 172.21.15.82:0/2938587666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09340ff3b0 msgr2=0x7f0934195290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0932ffd640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09340ff3b0 0x7f0934195290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0932ffd640 1 -- 172.21.15.82:0/2938587666 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0928009560 con 0x7f09340ffd60 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0932ffd640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 0x7f09341957d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0928006010 tx=0x7f09280055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0930ff9640 1 -- 172.21.15.82:0/2938587666 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0928004aa0 con 0x7f09340ffd60 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0930ff9640 1 -- 172.21.15.82:0/2938587666 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0928004c00 con 0x7f09340ffd60 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.093+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0934199e00 con 0x7f09340ffd60 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.094+0000 7f0930ff9640 1 -- 172.21.15.82:0/2938587666 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0928003a50 con 0x7f09340ffd60 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.094+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093419a2f0 con 0x7f09340ffd60 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.095+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0934100890 con 0x7f09340ffd60 2024-06-06T07:45:07.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.095+0000 7f0930ff9640 1 -- 172.21.15.82:0/2938587666 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0928003c90 con 0x7f09340ffd60 2024-06-06T07:45:07.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.095+0000 7f0930ff9640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f091c06af00 0x7f091c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:07.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.095+0000 7f0930ff9640 1 -- 172.21.15.82:0/2938587666 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f09280aae20 con 0x7f09340ffd60 2024-06-06T07:45:07.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.095+0000 7f09337fe640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f091c06af00 0x7f091c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:07.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.096+0000 7f09337fe640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f091c06af00 0x7f091c06d3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0924009890 tx=0x7f0924008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:07.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.100+0000 7f0930ff9640 1 -- 172.21.15.82:0/2938587666 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0928076520 con 0x7f09340ffd60 2024-06-06T07:45:07.306 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.306+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f093404e210 con 0x7f09340ffd60 2024-06-06T07:45:07.306 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.306+0000 7f0930ff9640 1 -- 172.21.15.82:0/2938587666 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0928079b40 con 0x7f09340ffd60 2024-06-06T07:45:07.306 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:07.306 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:07.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f091c06af00 msgr2=0x7f091c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:07.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f091c06af00 0x7f091c06d3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0924009890 tx=0x7f0924008040 comp rx=0 tx=0).stop 2024-06-06T07:45:07.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 msgr2=0x7f09341957d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:07.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 0x7f09341957d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0928006010 tx=0x7f09280055a0 comp rx=0 tx=0).stop 2024-06-06T07:45:07.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 shutdown_connections 2024-06-06T07:45:07.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f091c06af00 0x7f091c06d3c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:07.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09340ffd60 0x7f09341957d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:07.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 --2- 172.21.15.82:0/2938587666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09340ff3b0 0x7f0934195290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:07.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 >> 172.21.15.82:0/2938587666 conn(0x7f09340fac80 msgr2=0x7f09340fc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:07.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 shutdown_connections 2024-06-06T07:45:07.309 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:07.308+0000 7f0939ae5640 1 -- 172.21.15.82:0/2938587666 wait complete. 2024-06-06T07:45:07.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:45:07.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:45:07.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:45:07.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:07 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2938587666' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:07.949 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:45:08.055 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:45:08.055 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:45:08.055 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:45:08.055 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:07 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2938587666' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:08.055 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:45:09.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:08 smithi184 ceph-mon[37288]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:09.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:08 smithi082 ceph-mon[29354]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:10.971 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:11.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:10 smithi184 ceph-mon[37288]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:11.187 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:11.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:10 smithi082 ceph-mon[29354]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.779+0000 7f7209c77640 1 -- 172.21.15.82:0/3160294742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 msgr2=0x7f72040fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.779+0000 7f7209c77640 1 --2- 172.21.15.82:0/3160294742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 0x7f72040fee30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f71ec009880 tx=0x7f71ec02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.779+0000 7f7209c77640 1 -- 172.21.15.82:0/3160294742 shutdown_connections 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.779+0000 7f7209c77640 1 --2- 172.21.15.82:0/3160294742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 0x7f72040fee30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.779+0000 7f7209c77640 1 --2- 172.21.15.82:0/3160294742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72041062f0 0x7f72041066d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.779+0000 7f7209c77640 1 -- 172.21.15.82:0/3160294742 >> 172.21.15.82:0/3160294742 conn(0x7f7204074430 msgr2=0x7f7204074840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.779+0000 7f7209c77640 1 -- 172.21.15.82:0/3160294742 shutdown_connections 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.779+0000 7f7209c77640 1 -- 172.21.15.82:0/3160294742 wait complete. 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f7209c77640 1 Processor -- start 2024-06-06T07:45:12.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f7209c77640 1 -- start start 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f7209c77640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 0x7f72040ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f7209c77640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72041062f0 0x7f7204100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f7209c77640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72041018c0 con 0x7f72040fe9f0 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f7209c77640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7204101a00 con 0x7f72041062f0 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f72037fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 0x7f72040ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f72037fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 0x7f72040ffbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:54936/0 (socket says 172.21.15.82:54936) 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f72037fe640 1 -- 172.21.15.82:0/2537691954 learned_addr learned my addr 172.21.15.82:0/2537691954 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f72037fe640 1 -- 172.21.15.82:0/2537691954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72041062f0 msgr2=0x7f7204100110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f72037fe640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72041062f0 0x7f7204100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f72037fe640 1 -- 172.21.15.82:0/2537691954 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec009560 con 0x7f72040fe9f0 2024-06-06T07:45:12.809 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f72037fe640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 0x7f72040ffbd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f71f400b420 tx=0x7f71f400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f72017fa640 1 -- 172.21.15.82:0/2537691954 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71f4011800 con 0x7f72040fe9f0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f72017fa640 1 -- 172.21.15.82:0/2537691954 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71f4011e20 con 0x7f72040fe9f0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.780+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7204100650 con 0x7f72040fe9f0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.781+0000 7f72017fa640 1 -- 172.21.15.82:0/2537691954 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71f400f2e0 con 0x7f72040fe9f0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.781+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72041a6cc0 con 0x7f72040fe9f0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.782+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7204068410 con 0x7f72040fe9f0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.782+0000 7f72017fa640 1 -- 172.21.15.82:0/2537691954 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f71f4010380 con 0x7f72040fe9f0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.782+0000 7f72017fa640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f71f006aef0 0x7f71f006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.783+0000 7f71fbfff640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f71f006aef0 0x7f71f006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.783+0000 7f72017fa640 1 -- 172.21.15.82:0/2537691954 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f71f4089570 con 0x7f72040fe9f0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.783+0000 7f71fbfff640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f71f006aef0 0x7f71f006d3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7204101300 tx=0x7f71ec000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:12.810 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.787+0000 7f72017fa640 1 -- 172.21.15.82:0/2537691954 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f71f4054b50 con 0x7f72040fe9f0 2024-06-06T07:45:12.894 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:12 smithi082 ceph-mon[29354]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:12.998 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.998+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f720404e210 con 0x7f72040fe9f0 2024-06-06T07:45:12.998 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:12.998+0000 7f72017fa640 1 -- 172.21.15.82:0/2537691954 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f71f4058170 con 0x7f72040fe9f0 2024-06-06T07:45:12.998 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:12.999 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:13.000 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f71f006aef0 msgr2=0x7f71f006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:13.000 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f71f006aef0 0x7f71f006d3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7204101300 tx=0x7f71ec000bc0 comp rx=0 tx=0).stop 2024-06-06T07:45:13.001 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 msgr2=0x7f72040ffbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:13.001 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 0x7f72040ffbd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f71f400b420 tx=0x7f71f400b8e0 comp rx=0 tx=0).stop 2024-06-06T07:45:13.001 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 shutdown_connections 2024-06-06T07:45:13.001 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f71f006aef0 0x7f71f006d3b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:13.001 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72041062f0 0x7f7204100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:13.001 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 --2- 172.21.15.82:0/2537691954 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72040fe9f0 0x7f72040ffbd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:13.001 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 >> 172.21.15.82:0/2537691954 conn(0x7f7204074430 msgr2=0x7f72040fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:13.001 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.000+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 shutdown_connections 2024-06-06T07:45:13.001 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:13.001+0000 7f7209c77640 1 -- 172.21.15.82:0/2537691954 wait complete. 2024-06-06T07:45:13.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:12 smithi184 ceph-mon[37288]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:14.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:13 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2537691954' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:14.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:13 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2537691954' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:15.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:14 smithi184 ceph-mon[37288]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:15.169 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:14 smithi082 ceph-mon[29354]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:16.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:15 smithi184 ceph-mon[37288]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:16.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:15 smithi082 ceph-mon[29354]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:16.642 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:16.847 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:18.420 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.416+0000 7f2aa2c52640 1 -- 172.21.15.82:0/522058815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 msgr2=0x7f2a9c103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:18.420 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.416+0000 7f2aa2c52640 1 --2- 172.21.15.82:0/522058815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 0x7f2a9c103d60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009880 tx=0x7f2a8402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.416+0000 7f2aa2c52640 1 -- 172.21.15.82:0/522058815 shutdown_connections 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.416+0000 7f2aa2c52640 1 --2- 172.21.15.82:0/522058815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 0x7f2a9c103d60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.416+0000 7f2aa2c52640 1 --2- 172.21.15.82:0/522058815 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a9c0ff360 0x7f2a9c0ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.416+0000 7f2aa2c52640 1 -- 172.21.15.82:0/522058815 >> 172.21.15.82:0/522058815 conn(0x7f2a9c075e10 msgr2=0x7f2a9c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.416+0000 7f2aa2c52640 1 -- 172.21.15.82:0/522058815 shutdown_connections 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.416+0000 7f2aa2c52640 1 -- 172.21.15.82:0/522058815 wait complete. 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.417+0000 7f2aa2c52640 1 Processor -- start 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.417+0000 7f2aa2c52640 1 -- start start 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.417+0000 7f2aa2c52640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a9c0ff360 0x7f2a9c1961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.417+0000 7f2aa2c52640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 0x7f2a9c196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.417+0000 7f2aa2c52640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c196db0 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.417+0000 7f2aa2c52640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c19aad0 con 0x7f2a9c0ff360 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.417+0000 7f2a9bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 0x7f2a9c196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:18.421 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2a9bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 0x7f2a9c196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47810/0 (socket says 172.21.15.82:47810) 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2a9bfff640 1 -- 172.21.15.82:0/2770174572 learned_addr learned my addr 172.21.15.82:0/2770174572 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2aa09ee640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a9c0ff360 0x7f2a9c1961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2a9bfff640 1 -- 172.21.15.82:0/2770174572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a9c0ff360 msgr2=0x7f2a9c1961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2a9bfff640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a9c0ff360 0x7f2a9c1961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2a9bfff640 1 -- 172.21.15.82:0/2770174572 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a84009560 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2a9bfff640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 0x7f2a9c196720 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2a8402a9c0 tx=0x7f2a840055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2a99ffb640 1 -- 172.21.15.82:0/2770174572 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a84004aa0 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9c19ad50 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2a99ffb640 1 -- 172.21.15.82:0/2770174572 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a84031470 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2a99ffb640 1 -- 172.21.15.82:0/2770174572 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a840039e0 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.418+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9c19b240 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.422 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.420+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9c100840 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.423 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.420+0000 7f2a99ffb640 1 -- 172.21.15.82:0/2770174572 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f2a84003be0 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.423 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.420+0000 7f2a99ffb640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a8806aef0 0x7f2a8806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:18.423 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.420+0000 7f2a99ffb640 1 -- 172.21.15.82:0/2770174572 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2a840aac30 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.423 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.420+0000 7f2aa09ee640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a8806aef0 0x7f2a8806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:18.423 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.421+0000 7f2aa09ee640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a8806aef0 0x7f2a8806d3b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c005fd0 tx=0x7f2a8c00b390 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:18.424 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.425+0000 7f2a99ffb640 1 -- 172.21.15.82:0/2770174572 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2a840762c0 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.634 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.634+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2a9c04e210 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.634 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.635+0000 7f2a99ffb640 1 -- 172.21.15.82:0/2770174572 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2a840798e0 con 0x7f2a9c0ffd10 2024-06-06T07:45:18.635 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:18.635 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a8806aef0 msgr2=0x7f2a8806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a8806aef0 0x7f2a8806d3b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c005fd0 tx=0x7f2a8c00b390 comp rx=0 tx=0).stop 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 msgr2=0x7f2a9c196720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 0x7f2a9c196720 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2a8402a9c0 tx=0x7f2a840055a0 comp rx=0 tx=0).stop 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 shutdown_connections 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f2a8806aef0 0x7f2a8806d3b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a9c0ffd10 0x7f2a9c196720 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 --2- 172.21.15.82:0/2770174572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a9c0ff360 0x7f2a9c1961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 >> 172.21.15.82:0/2770174572 conn(0x7f2a9c075e10 msgr2=0x7f2a9c067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:18.637 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 shutdown_connections 2024-06-06T07:45:18.638 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:18.637+0000 7f2aa2c52640 1 -- 172.21.15.82:0/2770174572 wait complete. 2024-06-06T07:45:18.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:18 smithi082 ceph-mon[29354]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:18.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:18 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2770174572' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:19.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:18 smithi184 ceph-mon[37288]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:19.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:18 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2770174572' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:21.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:20 smithi082 ceph-mon[29354]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:21.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:20 smithi184 ceph-mon[37288]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:22.329 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:22.544 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:22.967 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:22 smithi082 ceph-mon[29354]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:22.968 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:45:23.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:22 smithi184 ceph-mon[37288]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:23.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:45:24.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.111+0000 7fae3032a640 1 -- 172.21.15.82:0/929564067 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 msgr2=0x7fae280ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:24.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.111+0000 7fae3032a640 1 --2- 172.21.15.82:0/929564067 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 0x7fae280ff780 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fae20009880 tx=0x7fae2002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:45:24.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.111+0000 7fae3032a640 1 -- 172.21.15.82:0/929564067 shutdown_connections 2024-06-06T07:45:24.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.111+0000 7fae3032a640 1 --2- 172.21.15.82:0/929564067 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae280ffd50 0x7fae28103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:24.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.111+0000 7fae3032a640 1 --2- 172.21.15.82:0/929564067 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 0x7fae280ff780 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:24.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.111+0000 7fae3032a640 1 -- 172.21.15.82:0/929564067 >> 172.21.15.82:0/929564067 conn(0x7fae280fac70 msgr2=0x7fae280fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:24.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.111+0000 7fae3032a640 1 -- 172.21.15.82:0/929564067 shutdown_connections 2024-06-06T07:45:24.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.111+0000 7fae3032a640 1 -- 172.21.15.82:0/929564067 wait complete. 2024-06-06T07:45:24.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.112+0000 7fae3032a640 1 Processor -- start 2024-06-06T07:45:24.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.112+0000 7fae3032a640 1 -- start start 2024-06-06T07:45:24.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.112+0000 7fae3032a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 0x7fae28195180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:24.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.112+0000 7fae3032a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae280ffd50 0x7fae281956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:24.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.112+0000 7fae3032a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae28195d50 con 0x7fae280ff3a0 2024-06-06T07:45:24.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.112+0000 7fae3032a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae28199a70 con 0x7fae280ffd50 2024-06-06T07:45:24.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.112+0000 7fae2e0c6640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 0x7fae28195180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:24.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.112+0000 7fae2e0c6640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 0x7fae28195180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47826/0 (socket says 172.21.15.82:47826) 2024-06-06T07:45:24.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.112+0000 7fae2e0c6640 1 -- 172.21.15.82:0/1172553040 learned_addr learned my addr 172.21.15.82:0/1172553040 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.113+0000 7fae2e0c6640 1 -- 172.21.15.82:0/1172553040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae280ffd50 msgr2=0x7fae281956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.113+0000 7fae2e0c6640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae280ffd50 0x7fae281956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.113+0000 7fae2e0c6640 1 -- 172.21.15.82:0/1172553040 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae20009560 con 0x7fae280ff3a0 2024-06-06T07:45:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.113+0000 7fae2e0c6640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 0x7fae28195180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fae2002a9c0 tx=0x7fae200055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.113+0000 7fae277fe640 1 -- 172.21.15.82:0/1172553040 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae20004aa0 con 0x7fae280ff3a0 2024-06-06T07:45:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.113+0000 7fae277fe640 1 -- 172.21.15.82:0/1172553040 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae20004c00 con 0x7fae280ff3a0 2024-06-06T07:45:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.113+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae28199cf0 con 0x7fae280ff3a0 2024-06-06T07:45:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.113+0000 7fae277fe640 1 -- 172.21.15.82:0/1172553040 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae20003ab0 con 0x7fae280ff3a0 2024-06-06T07:45:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.113+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2819a1e0 con 0x7fae280ff3a0 2024-06-06T07:45:24.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.114+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae28100880 con 0x7fae280ff3a0 2024-06-06T07:45:24.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.115+0000 7fae277fe640 1 -- 172.21.15.82:0/1172553040 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fae20031d20 con 0x7fae280ff3a0 2024-06-06T07:45:24.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.115+0000 7fae277fe640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae1006aef0 0x7fae1006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:24.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.115+0000 7fae277fe640 1 -- 172.21.15.82:0/1172553040 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fae200aac10 con 0x7fae280ff3a0 2024-06-06T07:45:24.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.115+0000 7fae2d8c5640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae1006aef0 0x7fae1006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:24.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.115+0000 7fae2d8c5640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae1006aef0 0x7fae1006d3b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fae281966e0 tx=0x7fae1c005f70 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:24.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.119+0000 7fae277fe640 1 -- 172.21.15.82:0/1172553040 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fae20076270 con 0x7fae280ff3a0 2024-06-06T07:45:24.326 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.326+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fae2804e210 con 0x7fae280ff3a0 2024-06-06T07:45:24.326 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.326+0000 7fae277fe640 1 -- 172.21.15.82:0/1172553040 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fae20079890 con 0x7fae280ff3a0 2024-06-06T07:45:24.327 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:24.328 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.328+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae1006aef0 msgr2=0x7fae1006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:24.328 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.328+0000 7fae3032a640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae1006aef0 0x7fae1006d3b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fae281966e0 tx=0x7fae1c005f70 comp rx=0 tx=0).stop 2024-06-06T07:45:24.328 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.328+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 msgr2=0x7fae28195180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:24.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.328+0000 7fae3032a640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 0x7fae28195180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fae2002a9c0 tx=0x7fae200055a0 comp rx=0 tx=0).stop 2024-06-06T07:45:24.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.328+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 shutdown_connections 2024-06-06T07:45:24.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.328+0000 7fae3032a640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fae1006aef0 0x7fae1006d3b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:24.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.328+0000 7fae3032a640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae280ffd50 0x7fae281956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:24.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.328+0000 7fae3032a640 1 --2- 172.21.15.82:0/1172553040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae280ff3a0 0x7fae28195180 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:24.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.328+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 >> 172.21.15.82:0/1172553040 conn(0x7fae280fac70 msgr2=0x7fae280fbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:24.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.329+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 shutdown_connections 2024-06-06T07:45:24.329 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:24.329+0000 7fae3032a640 1 -- 172.21.15.82:0/1172553040 wait complete. 2024-06-06T07:45:24.933 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:24 smithi082 ceph-mon[29354]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:24.933 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:24 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/1172553040' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:25.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:24 smithi184 ceph-mon[37288]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:25.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:24 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/1172553040' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:26.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:26 smithi082 ceph-mon[29354]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:27.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:26 smithi184 ceph-mon[37288]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:27.935 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:28.151 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:29.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:28 smithi082 ceph-mon[29354]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:29.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:28 smithi184 ceph-mon[37288]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:29.769 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.749+0000 7f0222fcd640 1 -- 172.21.15.82:0/3046110519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 msgr2=0x7f021c0ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:29.769 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.749+0000 7f0222fcd640 1 --2- 172.21.15.82:0/3046110519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 0x7f021c0ff780 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0210009880 tx=0x7f021002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:45:29.769 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.749+0000 7f0222fcd640 1 -- 172.21.15.82:0/3046110519 shutdown_connections 2024-06-06T07:45:29.769 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.749+0000 7f0222fcd640 1 --2- 172.21.15.82:0/3046110519 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f021c0ffd50 0x7f021c103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:29.769 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.749+0000 7f0222fcd640 1 --2- 172.21.15.82:0/3046110519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 0x7f021c0ff780 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.749+0000 7f0222fcd640 1 -- 172.21.15.82:0/3046110519 >> 172.21.15.82:0/3046110519 conn(0x7f021c0fac70 msgr2=0x7f021c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.749+0000 7f0222fcd640 1 -- 172.21.15.82:0/3046110519 shutdown_connections 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.749+0000 7f0222fcd640 1 -- 172.21.15.82:0/3046110519 wait complete. 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0222fcd640 1 Processor -- start 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0222fcd640 1 -- start start 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0222fcd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 0x7f021c197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0222fcd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f021c0ffd50 0x7f021c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0222fcd640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c197ad0 con 0x7f021c0ff3a0 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0222fcd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c197c40 con 0x7f021c0ffd50 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0220d69640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 0x7f021c197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0220d69640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 0x7f021c197050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:37972/0 (socket says 172.21.15.82:37972) 2024-06-06T07:45:29.770 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0220d69640 1 -- 172.21.15.82:0/3472573302 learned_addr learned my addr 172.21.15.82:0/3472573302 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f021bfff640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f021c0ffd50 0x7f021c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0220d69640 1 -- 172.21.15.82:0/3472573302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f021c0ffd50 msgr2=0x7f021c197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0220d69640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f021c0ffd50 0x7f021c197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0220d69640 1 -- 172.21.15.82:0/3472573302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210009560 con 0x7f021c0ff3a0 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f021bfff640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f021c0ffd50 0x7f021c197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.750+0000 7f0220d69640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 0x7f021c197050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f021002a9c0 tx=0x7f02100055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.751+0000 7f0219ffb640 1 -- 172.21.15.82:0/3472573302 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0210004aa0 con 0x7f021c0ff3a0 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.751+0000 7f0219ffb640 1 -- 172.21.15.82:0/3472573302 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0210004c00 con 0x7f021c0ff3a0 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.751+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c197ec0 con 0x7f021c0ff3a0 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.751+0000 7f0219ffb640 1 -- 172.21.15.82:0/3472573302 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0210003ab0 con 0x7f021c0ff3a0 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.751+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c1a03d0 con 0x7f021c0ff3a0 2024-06-06T07:45:29.771 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.752+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021c100880 con 0x7f021c0ff3a0 2024-06-06T07:45:29.772 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.752+0000 7f0219ffb640 1 -- 172.21.15.82:0/3472573302 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0210031d20 con 0x7f021c0ff3a0 2024-06-06T07:45:29.772 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.753+0000 7f0219ffb640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f020c06aef0 0x7f020c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:29.772 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.753+0000 7f0219ffb640 1 -- 172.21.15.82:0/3472573302 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f02100aaad0 con 0x7f021c0ff3a0 2024-06-06T07:45:29.772 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.753+0000 7f021bfff640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f020c06aef0 0x7f020c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:29.772 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.753+0000 7f021bfff640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f020c06aef0 0x7f020c06d3b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f021c068fd0 tx=0x7f020800a5c0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:29.772 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.757+0000 7f0219ffb640 1 -- 172.21.15.82:0/3472573302 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0210076130 con 0x7f021c0ff3a0 2024-06-06T07:45:29.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.965+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f021c197ec0 con 0x7f021c0ff3a0 2024-06-06T07:45:29.966 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.966+0000 7f0219ffb640 1 -- 172.21.15.82:0/3472573302 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f021c197ec0 con 0x7f021c0ff3a0 2024-06-06T07:45:29.966 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:29.966 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:29.969 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f020c06aef0 msgr2=0x7f020c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:29.969 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f020c06aef0 0x7f020c06d3b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f021c068fd0 tx=0x7f020800a5c0 comp rx=0 tx=0).stop 2024-06-06T07:45:29.969 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 msgr2=0x7f021c197050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:29.969 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 0x7f021c197050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f021002a9c0 tx=0x7f02100055a0 comp rx=0 tx=0).stop 2024-06-06T07:45:29.969 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 shutdown_connections 2024-06-06T07:45:29.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f020c06aef0 0x7f020c06d3b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:29.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f021c0ffd50 0x7f021c197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:29.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 --2- 172.21.15.82:0/3472573302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f021c0ff3a0 0x7f021c197050 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:29.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 >> 172.21.15.82:0/3472573302 conn(0x7f021c0fac70 msgr2=0x7f021c0fbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:29.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 shutdown_connections 2024-06-06T07:45:29.970 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:29.969+0000 7f0222fcd640 1 -- 172.21.15.82:0/3472573302 wait complete. 2024-06-06T07:45:31.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:30 smithi082 ceph-mon[29354]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:31.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:30 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3472573302' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:31.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:30 smithi184 ceph-mon[37288]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:31.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:30 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3472573302' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:33.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:32 smithi082 ceph-mon[29354]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:33.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:32 smithi184 ceph-mon[37288]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:33.546 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:33.762 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:35.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:34 smithi082 ceph-mon[29354]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:35.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:34 smithi184 ceph-mon[37288]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:35.387 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.351+0000 7fe049744640 1 -- 172.21.15.82:0/2672985741 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101d60 msgr2=0x7fe04410a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:35.387 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.351+0000 7fe049744640 1 --2- 172.21.15.82:0/2672985741 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101d60 0x7fe04410a250 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe02c009880 tx=0x7fe02c02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:45:35.387 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.351+0000 7fe049744640 1 -- 172.21.15.82:0/2672985741 shutdown_connections 2024-06-06T07:45:35.387 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.351+0000 7fe049744640 1 --2- 172.21.15.82:0/2672985741 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101d60 0x7fe04410a250 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:35.387 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.351+0000 7fe049744640 1 --2- 172.21.15.82:0/2672985741 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe044101440 0x7fe044101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:35.387 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.351+0000 7fe049744640 1 -- 172.21.15.82:0/2672985741 >> 172.21.15.82:0/2672985741 conn(0x7fe0440fac30 msgr2=0x7fe0440fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:35.387 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.351+0000 7fe049744640 1 -- 172.21.15.82:0/2672985741 shutdown_connections 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.351+0000 7fe049744640 1 -- 172.21.15.82:0/2672985741 wait complete. 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.352+0000 7fe049744640 1 Processor -- start 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.352+0000 7fe049744640 1 -- start start 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.352+0000 7fe049744640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101440 0x7fe0441a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.352+0000 7fe049744640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe044101d60 0x7fe0441a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.352+0000 7fe049744640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04419c660 con 0x7fe044101440 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.352+0000 7fe049744640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04419c7a0 con 0x7fe044101d60 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.352+0000 7fe042ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101440 0x7fe0441a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.352+0000 7fe042ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101440 0x7fe0441a25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:43350/0 (socket says 172.21.15.82:43350) 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.352+0000 7fe042ffd640 1 -- 172.21.15.82:0/3687132380 learned_addr learned my addr 172.21.15.82:0/3687132380 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe03bfff640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe044101d60 0x7fe0441a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:35.388 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe042ffd640 1 -- 172.21.15.82:0/3687132380 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe044101d60 msgr2=0x7fe0441a2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe042ffd640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe044101d60 0x7fe0441a2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe042ffd640 1 -- 172.21.15.82:0/3687132380 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c009560 con 0x7fe044101440 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe042ffd640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101440 0x7fe0441a25b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe03400d6f0 tx=0x7fe03400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe040ff9640 1 -- 172.21.15.82:0/3687132380 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe034009a30 con 0x7fe044101440 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe040ff9640 1 -- 172.21.15.82:0/3687132380 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0340054e0 con 0x7fe044101440 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe040ff9640 1 -- 172.21.15.82:0/3687132380 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe034010460 con 0x7fe044101440 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04419ca80 con 0x7fe044101440 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.353+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04419cfd0 con 0x7fe044101440 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.354+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe044068d40 con 0x7fe044101440 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.354+0000 7fe040ff9640 1 -- 172.21.15.82:0/3687132380 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fe034009b90 con 0x7fe044101440 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.355+0000 7fe040ff9640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe03006aef0 0x7fe03006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.355+0000 7fe040ff9640 1 -- 172.21.15.82:0/3687132380 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe0340896b0 con 0x7fe044101440 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.355+0000 7fe03bfff640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe03006aef0 0x7fe03006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:35.389 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.355+0000 7fe03bfff640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe03006aef0 0x7fe03006d3b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe02c000c00 tx=0x7fe02c004040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:35.390 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.359+0000 7fe040ff9640 1 -- 172.21.15.82:0/3687132380 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe034054d10 con 0x7fe044101440 2024-06-06T07:45:35.569 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.568+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe04404e210 con 0x7fe044101440 2024-06-06T07:45:35.569 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.568+0000 7fe040ff9640 1 -- 172.21.15.82:0/3687132380 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe0340200d0 con 0x7fe044101440 2024-06-06T07:45:35.569 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:35.569 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:35.571 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe03006aef0 msgr2=0x7fe03006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:35.571 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe03006aef0 0x7fe03006d3b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe02c000c00 tx=0x7fe02c004040 comp rx=0 tx=0).stop 2024-06-06T07:45:35.571 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101440 msgr2=0x7fe0441a25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:35.571 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101440 0x7fe0441a25b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe03400d6f0 tx=0x7fe03400dbb0 comp rx=0 tx=0).stop 2024-06-06T07:45:35.571 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 shutdown_connections 2024-06-06T07:45:35.571 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe03006aef0 0x7fe03006d3b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:35.572 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe044101d60 0x7fe0441a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:35.572 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 --2- 172.21.15.82:0/3687132380 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe044101440 0x7fe0441a25b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:35.572 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 >> 172.21.15.82:0/3687132380 conn(0x7fe0440fac30 msgr2=0x7fe0440ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:35.572 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 shutdown_connections 2024-06-06T07:45:35.572 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:35.570+0000 7fe049744640 1 -- 172.21.15.82:0/3687132380 wait complete. 2024-06-06T07:45:36.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:35 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3687132380' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:36.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:35 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3687132380' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:36.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:36 smithi082 ceph-mon[29354]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:37.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:36 smithi184 ceph-mon[37288]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:38.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:45:38.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:45:39.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:38 smithi082 ceph-mon[29354]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:39.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:38 smithi184 ceph-mon[37288]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:39.233 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:39.452 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:40.877 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:40 smithi082 ceph-mon[29354]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:41.094 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.088+0000 7f9b24fde640 1 -- 172.21.15.82:0/986207434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ffd50 msgr2=0x7f9b20103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:41.094 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.088+0000 7f9b24fde640 1 --2- 172.21.15.82:0/986207434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ffd50 0x7f9b20103da0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9b08009880 tx=0x7f9b0802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:45:41.094 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.088+0000 7f9b24fde640 1 -- 172.21.15.82:0/986207434 shutdown_connections 2024-06-06T07:45:41.094 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.088+0000 7f9b24fde640 1 --2- 172.21.15.82:0/986207434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ffd50 0x7f9b20103da0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:41.094 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.088+0000 7f9b24fde640 1 --2- 172.21.15.82:0/986207434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b200ff3a0 0x7f9b200ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:41.094 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.088+0000 7f9b24fde640 1 -- 172.21.15.82:0/986207434 >> 172.21.15.82:0/986207434 conn(0x7f9b200fac50 msgr2=0x7f9b200fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:41.094 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.088+0000 7f9b24fde640 1 -- 172.21.15.82:0/986207434 shutdown_connections 2024-06-06T07:45:41.094 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.089+0000 7f9b24fde640 1 -- 172.21.15.82:0/986207434 wait complete. 2024-06-06T07:45:41.094 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.089+0000 7f9b24fde640 1 Processor -- start 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.089+0000 7f9b24fde640 1 -- start start 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.089+0000 7f9b24fde640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ff3a0 0x7f9b20196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.089+0000 7f9b24fde640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b200ffd50 0x7f9b201967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.089+0000 7f9b24fde640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b20196e40 con 0x7f9b200ff3a0 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.089+0000 7f9b24fde640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2019ab60 con 0x7f9b200ffd50 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b1e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ff3a0 0x7f9b20196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b1e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ff3a0 0x7f9b20196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:43356/0 (socket says 172.21.15.82:43356) 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b1e59c640 1 -- 172.21.15.82:0/875091690 learned_addr learned my addr 172.21.15.82:0/875091690 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b1dd9b640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b200ffd50 0x7f9b201967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:41.095 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b1e59c640 1 -- 172.21.15.82:0/875091690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b200ffd50 msgr2=0x7f9b201967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b1e59c640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b200ffd50 0x7f9b201967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b1e59c640 1 -- 172.21.15.82:0/875091690 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b08009560 con 0x7f9b200ff3a0 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b1e59c640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ff3a0 0x7f9b20196270 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800d6f0 tx=0x7f9b1800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b177fe640 1 -- 172.21.15.82:0/875091690 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b18009a30 con 0x7f9b200ff3a0 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2019ae40 con 0x7f9b200ff3a0 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b177fe640 1 -- 172.21.15.82:0/875091690 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b180054e0 con 0x7f9b200ff3a0 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b177fe640 1 -- 172.21.15.82:0/875091690 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b18010460 con 0x7f9b200ff3a0 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.090+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2019b360 con 0x7f9b200ff3a0 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.091+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b20100880 con 0x7f9b200ff3a0 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.092+0000 7f9b177fe640 1 -- 172.21.15.82:0/875091690 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f9b18009b90 con 0x7f9b200ff3a0 2024-06-06T07:45:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.092+0000 7f9b177fe640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9b0c06aef0 0x7f9b0c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:41.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.092+0000 7f9b177fe640 1 -- 172.21.15.82:0/875091690 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9b180897a0 con 0x7f9b200ff3a0 2024-06-06T07:45:41.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.092+0000 7f9b1dd9b640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9b0c06aef0 0x7f9b0c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:41.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.093+0000 7f9b1dd9b640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9b0c06aef0 0x7f9b0c06d3b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b08009230 tx=0x7f9b08005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:41.097 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:40 smithi184 ceph-mon[37288]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:41.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.096+0000 7f9b177fe640 1 -- 172.21.15.82:0/875091690 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9b18054d80 con 0x7f9b200ff3a0 2024-06-06T07:45:41.307 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.306+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9b2004e210 con 0x7f9b200ff3a0 2024-06-06T07:45:41.308 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.307+0000 7f9b177fe640 1 -- 172.21.15.82:0/875091690 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9b180200d0 con 0x7f9b200ff3a0 2024-06-06T07:45:41.308 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:41.308 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.308+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9b0c06aef0 msgr2=0x7f9b0c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.308+0000 7f9b24fde640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9b0c06aef0 0x7f9b0c06d3b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b08009230 tx=0x7f9b08005a20 comp rx=0 tx=0).stop 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.308+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ff3a0 msgr2=0x7f9b20196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.308+0000 7f9b24fde640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ff3a0 0x7f9b20196270 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800d6f0 tx=0x7f9b1800dbb0 comp rx=0 tx=0).stop 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.308+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 shutdown_connections 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.308+0000 7f9b24fde640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9b0c06aef0 0x7f9b0c06d3b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.308+0000 7f9b24fde640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b200ffd50 0x7f9b201967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.308+0000 7f9b24fde640 1 --2- 172.21.15.82:0/875091690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b200ff3a0 0x7f9b20196270 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.308+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 >> 172.21.15.82:0/875091690 conn(0x7f9b200fac50 msgr2=0x7f9b20109520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.309+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 shutdown_connections 2024-06-06T07:45:41.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:41.309+0000 7f9b24fde640 1 -- 172.21.15.82:0/875091690 wait complete. 2024-06-06T07:45:41.880 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:41 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/875091690' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:42.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:41 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/875091690' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:43.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:42 smithi082 ceph-mon[29354]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:43.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:42 smithi184 ceph-mon[37288]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:44.902 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:45.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:44 smithi082 ceph-mon[29354]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:45.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:44 smithi184 ceph-mon[37288]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:45.120 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:46.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 -- 172.21.15.82:0/3934971385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101480 msgr2=0x7f6a58101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:46.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 --2- 172.21.15.82:0/3934971385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101480 0x7f6a58101860 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009880 tx=0x7f6a5402a520 comp rx=0 tx=0).stop 2024-06-06T07:45:46.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 -- 172.21.15.82:0/3934971385 shutdown_connections 2024-06-06T07:45:46.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 --2- 172.21.15.82:0/3934971385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a58101da0 0x7f6a5810a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:46.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 --2- 172.21.15.82:0/3934971385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101480 0x7f6a58101860 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:46.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 -- 172.21.15.82:0/3934971385 >> 172.21.15.82:0/3934971385 conn(0x7f6a58075e10 msgr2=0x7f6a580fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:46.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 -- 172.21.15.82:0/3934971385 shutdown_connections 2024-06-06T07:45:46.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 -- 172.21.15.82:0/3934971385 wait complete. 2024-06-06T07:45:46.617 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 Processor -- start 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.615+0000 7f6a5f34f640 1 -- start start 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5f34f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a58101480 0x7f6a580ff0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5f34f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101da0 0x7f6a580ff620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5f34f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a580ffb60 con 0x7f6a58101da0 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5f34f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a580ffcd0 con 0x7f6a58101480 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5c8ea640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101da0 0x7f6a580ff620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5c8ea640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101da0 0x7f6a580ff620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:58176/0 (socket says 172.21.15.82:58176) 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5c8ea640 1 -- 172.21.15.82:0/2624112753 learned_addr learned my addr 172.21.15.82:0/2624112753 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5d0eb640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a58101480 0x7f6a580ff0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5c8ea640 1 -- 172.21.15.82:0/2624112753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a58101480 msgr2=0x7f6a580ff0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5c8ea640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a58101480 0x7f6a580ff0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:46.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5c8ea640 1 -- 172.21.15.82:0/2624112753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a54009560 con 0x7f6a58101da0 2024-06-06T07:45:46.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5c8ea640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101da0 0x7f6a580ff620 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800d8d0 tx=0x7f6a4800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:46.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a4e7fc640 1 -- 172.21.15.82:0/2624112753 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a4800bea0 con 0x7f6a58101da0 2024-06-06T07:45:46.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a581a2430 con 0x7f6a58101da0 2024-06-06T07:45:46.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a4e7fc640 1 -- 172.21.15.82:0/2624112753 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a48009c00 con 0x7f6a58101da0 2024-06-06T07:45:46.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.616+0000 7f6a4e7fc640 1 -- 172.21.15.82:0/2624112753 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a4800f710 con 0x7f6a58101da0 2024-06-06T07:45:46.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.617+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a581a2790 con 0x7f6a58101da0 2024-06-06T07:45:46.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.618+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a58068450 con 0x7f6a58101da0 2024-06-06T07:45:46.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.618+0000 7f6a4e7fc640 1 -- 172.21.15.82:0/2624112753 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f6a4800f990 con 0x7f6a58101da0 2024-06-06T07:45:46.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.618+0000 7f6a4e7fc640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6a4406aef0 0x7f6a4406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:46.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.618+0000 7f6a4e7fc640 1 -- 172.21.15.82:0/2624112753 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6a48089970 con 0x7f6a58101da0 2024-06-06T07:45:46.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.618+0000 7f6a5d0eb640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6a4406aef0 0x7f6a4406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:46.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.619+0000 7f6a5d0eb640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6a4406aef0 0x7f6a4406d3b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6a54000c00 tx=0x7f6a54005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:46.622 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.623+0000 7f6a4e7fc640 1 -- 172.21.15.82:0/2624112753 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6a48054f50 con 0x7f6a58101da0 2024-06-06T07:45:46.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.835+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6a58104b40 con 0x7f6a58101da0 2024-06-06T07:45:46.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.836+0000 7f6a4e7fc640 1 -- 172.21.15.82:0/2624112753 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6a48058570 con 0x7f6a58101da0 2024-06-06T07:45:46.836 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:46.836 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6a4406aef0 msgr2=0x7f6a4406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6a4406aef0 0x7f6a4406d3b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6a54000c00 tx=0x7f6a54005a20 comp rx=0 tx=0).stop 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101da0 msgr2=0x7f6a580ff620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101da0 0x7f6a580ff620 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800d8d0 tx=0x7f6a4800dd90 comp rx=0 tx=0).stop 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 shutdown_connections 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f6a4406aef0 0x7f6a4406d3b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a58101da0 0x7f6a580ff620 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 --2- 172.21.15.82:0/2624112753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a58101480 0x7f6a580ff0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 >> 172.21.15.82:0/2624112753 conn(0x7f6a58075e10 msgr2=0x7f6a58102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 shutdown_connections 2024-06-06T07:45:46.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:46.838+0000 7f6a5f34f640 1 -- 172.21.15.82:0/2624112753 wait complete. 2024-06-06T07:45:47.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:46 smithi082 ceph-mon[29354]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:47.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:46 smithi184 ceph-mon[37288]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:48.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:47 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2624112753' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:48.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:47 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2624112753' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:48.843 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:48 smithi184 ceph-mon[37288]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:49.011 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:48 smithi082 ceph-mon[29354]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:50.483 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:50.702 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:50.770 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:50 smithi082 ceph-mon[29354]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:51.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:50 smithi184 ceph-mon[37288]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.213+0000 7f15eda04640 1 -- 172.21.15.82:0/3734269407 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ffd70 msgr2=0x7f15e8103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.213+0000 7f15eda04640 1 --2- 172.21.15.82:0/3734269407 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ffd70 0x7f15e8103dc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009c20 tx=0x7f15dc02a970 comp rx=0 tx=0).stop 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.213+0000 7f15eda04640 1 -- 172.21.15.82:0/3734269407 shutdown_connections 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.213+0000 7f15eda04640 1 --2- 172.21.15.82:0/3734269407 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ffd70 0x7f15e8103dc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.213+0000 7f15eda04640 1 --2- 172.21.15.82:0/3734269407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e80ff3c0 0x7f15e80ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.213+0000 7f15eda04640 1 -- 172.21.15.82:0/3734269407 >> 172.21.15.82:0/3734269407 conn(0x7f15e80fac70 msgr2=0x7f15e80fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.213+0000 7f15eda04640 1 -- 172.21.15.82:0/3734269407 shutdown_connections 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.213+0000 7f15eda04640 1 -- 172.21.15.82:0/3734269407 wait complete. 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.213+0000 7f15eda04640 1 Processor -- start 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15eda04640 1 -- start start 2024-06-06T07:45:52.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15eda04640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ff3c0 0x7f15e8195180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15eda04640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e80ffd70 0x7f15e81956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15eda04640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8195d50 con 0x7f15e80ff3c0 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15eda04640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8199a70 con 0x7f15e80ffd70 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15e6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ff3c0 0x7f15e8195180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15e6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ff3c0 0x7f15e8195180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:58190/0 (socket says 172.21.15.82:58190) 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15e6ffd640 1 -- 172.21.15.82:0/2517870253 learned_addr learned my addr 172.21.15.82:0/2517870253 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15e67fc640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e80ffd70 0x7f15e81956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15e6ffd640 1 -- 172.21.15.82:0/2517870253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e80ffd70 msgr2=0x7f15e81956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15e6ffd640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e80ffd70 0x7f15e81956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:52.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15e6ffd640 1 -- 172.21.15.82:0/2517870253 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc009900 con 0x7f15e80ff3c0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15e67fc640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e80ffd70 0x7f15e81956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.214+0000 7f15e6ffd640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ff3c0 0x7f15e8195180 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f15e000c8f0 tx=0x7f15e000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.215+0000 7f15eca02640 1 -- 172.21.15.82:0/2517870253 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15e0007b80 con 0x7f15e80ff3c0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.215+0000 7f15eca02640 1 -- 172.21.15.82:0/2517870253 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15e00054e0 con 0x7f15e80ff3c0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.215+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e8199d50 con 0x7f15e80ff3c0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.215+0000 7f15eca02640 1 -- 172.21.15.82:0/2517870253 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15e000f450 con 0x7f15e80ff3c0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.215+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e819a270 con 0x7f15e80ff3c0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.216+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e81008a0 con 0x7f15e80ff3c0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.216+0000 7f15eca02640 1 -- 172.21.15.82:0/2517870253 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f15e0004f60 con 0x7f15e80ff3c0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.217+0000 7f15eca02640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15d006aef0 0x7f15d006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.217+0000 7f15e67fc640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15d006aef0 0x7f15d006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.217+0000 7f15eca02640 1 -- 172.21.15.82:0/2517870253 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f15e0089a60 con 0x7f15e80ff3c0 2024-06-06T07:45:52.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.217+0000 7f15e67fc640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15d006aef0 0x7f15d006d3b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f15dc000f80 tx=0x7f15dc009860 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:52.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.221+0000 7f15eca02640 1 -- 172.21.15.82:0/2517870253 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f15e0055040 con 0x7f15e80ff3c0 2024-06-06T07:45:52.430 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.429+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f15e804e210 con 0x7f15e80ff3c0 2024-06-06T07:45:52.430 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.430+0000 7f15eca02640 1 -- 172.21.15.82:0/2517870253 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f15e001d0d0 con 0x7f15e80ff3c0 2024-06-06T07:45:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:52.431 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:52.432 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.431+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15d006aef0 msgr2=0x7f15d006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:52.432 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.431+0000 7f15eda04640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15d006aef0 0x7f15d006d3b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f15dc000f80 tx=0x7f15dc009860 comp rx=0 tx=0).stop 2024-06-06T07:45:52.432 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.431+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ff3c0 msgr2=0x7f15e8195180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:52.432 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.431+0000 7f15eda04640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ff3c0 0x7f15e8195180 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f15e000c8f0 tx=0x7f15e000cdb0 comp rx=0 tx=0).stop 2024-06-06T07:45:52.432 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.432+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 shutdown_connections 2024-06-06T07:45:52.432 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.432+0000 7f15eda04640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15d006aef0 0x7f15d006d3b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:52.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.432+0000 7f15eda04640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15e80ffd70 0x7f15e81956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:52.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.432+0000 7f15eda04640 1 --2- 172.21.15.82:0/2517870253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15e80ff3c0 0x7f15e8195180 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:52.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.432+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 >> 172.21.15.82:0/2517870253 conn(0x7f15e80fac70 msgr2=0x7f15e8102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:52.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.432+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 shutdown_connections 2024-06-06T07:45:52.433 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:52.432+0000 7f15eda04640 1 -- 172.21.15.82:0/2517870253 wait complete. 2024-06-06T07:45:53.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:52 smithi082 ceph-mon[29354]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:53.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:52 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2517870253' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:53.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:45:53.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:52 smithi184 ceph-mon[37288]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:52 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2517870253' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:53.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:45:55.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:54 smithi082 ceph-mon[29354]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:55.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:54 smithi184 ceph-mon[37288]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:56.100 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:45:56.317 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:45:57.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:56 smithi082 ceph-mon[29354]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:57.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:56 smithi184 ceph-mon[37288]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:57.929 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.918+0000 7fe34eacd640 1 -- 172.21.15.82:0/872686869 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 msgr2=0x7fe348104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:57.930 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.918+0000 7fe34eacd640 1 --2- 172.21.15.82:0/872686869 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 0x7fe348104ea0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009880 tx=0x7fe33c02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:45:57.930 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.918+0000 7fe34eacd640 1 -- 172.21.15.82:0/872686869 shutdown_connections 2024-06-06T07:45:57.930 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.918+0000 7fe34eacd640 1 --2- 172.21.15.82:0/872686869 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 0x7fe348104ea0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:57.930 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.918+0000 7fe34eacd640 1 --2- 172.21.15.82:0/872686869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3480feca0 0x7fe3480ff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:57.930 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.918+0000 7fe34eacd640 1 -- 172.21.15.82:0/872686869 >> 172.21.15.82:0/872686869 conn(0x7fe348074460 msgr2=0x7fe348074870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:57.930 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.918+0000 7fe34eacd640 1 -- 172.21.15.82:0/872686869 shutdown_connections 2024-06-06T07:45:57.934 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.918+0000 7fe34eacd640 1 -- 172.21.15.82:0/872686869 wait complete. 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe34eacd640 1 Processor -- start 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe34eacd640 1 -- start start 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe34eacd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3480feca0 0x7fe348196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe34eacd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 0x7fe348196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe34eacd640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe348196dd0 con 0x7fe3480ff650 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe34eacd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34819aaf0 con 0x7fe3480feca0 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe347fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 0x7fe348196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe347fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 0x7fe348196740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57126/0 (socket says 172.21.15.82:57126) 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe347fff640 1 -- 172.21.15.82:0/3300430037 learned_addr learned my addr 172.21.15.82:0/3300430037 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:45:57.935 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe347fff640 1 -- 172.21.15.82:0/3300430037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3480feca0 msgr2=0x7fe348196200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe347fff640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3480feca0 0x7fe348196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe347fff640 1 -- 172.21.15.82:0/3300430037 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c009560 con 0x7fe3480ff650 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe347fff640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 0x7fe348196740 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe33c02a9c0 tx=0x7fe33c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe345ffb640 1 -- 172.21.15.82:0/3300430037 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33c004aa0 con 0x7fe3480ff650 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe345ffb640 1 -- 172.21.15.82:0/3300430037 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe33c004c00 con 0x7fe3480ff650 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.919+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34819ad70 con 0x7fe3480ff650 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.920+0000 7fe345ffb640 1 -- 172.21.15.82:0/3300430037 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33c003a50 con 0x7fe3480ff650 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.920+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34819b2e0 con 0x7fe3480ff650 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.921+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3480683f0 con 0x7fe3480ff650 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.921+0000 7fe345ffb640 1 -- 172.21.15.82:0/3300430037 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fe33c031c90 con 0x7fe3480ff650 2024-06-06T07:45:57.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.922+0000 7fe345ffb640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe33006aef0 0x7fe33006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:45:57.937 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.922+0000 7fe345ffb640 1 -- 172.21.15.82:0/3300430037 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe33c0aadf0 con 0x7fe3480ff650 2024-06-06T07:45:57.937 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.922+0000 7fe34c869640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe33006aef0 0x7fe33006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:45:57.937 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.922+0000 7fe34c869640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe33006aef0 0x7fe33006d3b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe338007be0 tx=0x7fe338005f70 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:45:57.937 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:57.926+0000 7fe345ffb640 1 -- 172.21.15.82:0/3300430037 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe33c0763d0 con 0x7fe3480ff650 2024-06-06T07:45:58.139 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.135+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe34804e210 con 0x7fe3480ff650 2024-06-06T07:45:58.139 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.135+0000 7fe345ffb640 1 -- 172.21.15.82:0/3300430037 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe33c0799f0 con 0x7fe3480ff650 2024-06-06T07:45:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:45:58.139 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:45:58.140 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe33006aef0 msgr2=0x7fe33006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:58.140 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe33006aef0 0x7fe33006d3b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe338007be0 tx=0x7fe338005f70 comp rx=0 tx=0).stop 2024-06-06T07:45:58.140 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 msgr2=0x7fe348196740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:45:58.140 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 0x7fe348196740 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe33c02a9c0 tx=0x7fe33c0055a0 comp rx=0 tx=0).stop 2024-06-06T07:45:58.140 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 shutdown_connections 2024-06-06T07:45:58.140 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fe33006aef0 0x7fe33006d3b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:58.140 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3480ff650 0x7fe348196740 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:58.140 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 --2- 172.21.15.82:0/3300430037 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3480feca0 0x7fe348196200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:45:58.140 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 >> 172.21.15.82:0/3300430037 conn(0x7fe348074460 msgr2=0x7fe3480fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:45:58.141 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 shutdown_connections 2024-06-06T07:45:58.141 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:45:58.137+0000 7fe34eacd640 1 -- 172.21.15.82:0/3300430037 wait complete. 2024-06-06T07:45:59.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:58 smithi082 ceph-mon[29354]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:59.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:45:58 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3300430037' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:45:59.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:58 smithi184 ceph-mon[37288]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:45:59.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:45:58 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3300430037' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:01.018 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:00 smithi082 ceph-mon[29354]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:01.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:00 smithi184 ceph-mon[37288]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:01.722 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:01.938 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:03.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:02 smithi082 ceph-mon[29354]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:03.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:02 smithi184 ceph-mon[37288]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:03.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.665+0000 7fd136ae6640 1 -- 172.21.15.82:0/2330257857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 msgr2=0x7fd130103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:03.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.665+0000 7fd136ae6640 1 --2- 172.21.15.82:0/2330257857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 0x7fd130103d60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd118009880 tx=0x7fd11802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:46:03.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.665+0000 7fd136ae6640 1 -- 172.21.15.82:0/2330257857 shutdown_connections 2024-06-06T07:46:03.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.665+0000 7fd136ae6640 1 --2- 172.21.15.82:0/2330257857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 0x7fd130103d60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:03.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.665+0000 7fd136ae6640 1 --2- 172.21.15.82:0/2330257857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1300ff360 0x7fd1300ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.665+0000 7fd136ae6640 1 -- 172.21.15.82:0/2330257857 >> 172.21.15.82:0/2330257857 conn(0x7fd130075e10 msgr2=0x7fd1300fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.665+0000 7fd136ae6640 1 -- 172.21.15.82:0/2330257857 shutdown_connections 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.665+0000 7fd136ae6640 1 -- 172.21.15.82:0/2330257857 wait complete. 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.665+0000 7fd136ae6640 1 Processor -- start 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd136ae6640 1 -- start start 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd136ae6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1300ff360 0x7fd130196230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd136ae6640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 0x7fd130196770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd136ae6640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd130196e00 con 0x7fd1300ffd10 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd136ae6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13019aad0 con 0x7fd1300ff360 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12ffff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 0x7fd130196770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12ffff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 0x7fd130196770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:57148/0 (socket says 172.21.15.82:57148) 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12ffff640 1 -- 172.21.15.82:0/649314727 learned_addr learned my addr 172.21.15.82:0/649314727 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:03.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd134882640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1300ff360 0x7fd130196230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12ffff640 1 -- 172.21.15.82:0/649314727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1300ff360 msgr2=0x7fd130196230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12ffff640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1300ff360 0x7fd130196230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12ffff640 1 -- 172.21.15.82:0/649314727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd118009560 con 0x7fd1300ffd10 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12ffff640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 0x7fd130196770 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd11802a9c0 tx=0x7fd1180055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12dffb640 1 -- 172.21.15.82:0/649314727 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd118004aa0 con 0x7fd1300ffd10 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12dffb640 1 -- 172.21.15.82:0/649314727 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd118031470 con 0x7fd1300ffd10 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd12dffb640 1 -- 172.21.15.82:0/649314727 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1180039e0 con 0x7fd1300ffd10 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.666+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13019ad50 con 0x7fd1300ffd10 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.667+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13019b1f0 con 0x7fd1300ffd10 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.668+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd130100840 con 0x7fd1300ffd10 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.668+0000 7fd12dffb640 1 -- 172.21.15.82:0/649314727 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fd118031c00 con 0x7fd1300ffd10 2024-06-06T07:46:03.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.668+0000 7fd12dffb640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd11c06aef0 0x7fd11c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:03.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.669+0000 7fd12dffb640 1 -- 172.21.15.82:0/649314727 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd1180aac10 con 0x7fd1300ffd10 2024-06-06T07:46:03.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.669+0000 7fd134882640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd11c06aef0 0x7fd11c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:03.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.669+0000 7fd134882640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd11c06aef0 0x7fd11c06d3b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd120005fd0 tx=0x7fd120009680 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:03.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.673+0000 7fd12dffb640 1 -- 172.21.15.82:0/649314727 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd1180762a0 con 0x7fd1300ffd10 2024-06-06T07:46:03.881 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.881+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd13004e210 con 0x7fd1300ffd10 2024-06-06T07:46:03.881 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.882+0000 7fd12dffb640 1 -- 172.21.15.82:0/649314727 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd1180798c0 con 0x7fd1300ffd10 2024-06-06T07:46:03.882 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:03.882 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:03.883 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.883+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd11c06aef0 msgr2=0x7fd11c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd11c06aef0 0x7fd11c06d3b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd120005fd0 tx=0x7fd120009680 comp rx=0 tx=0).stop 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 msgr2=0x7fd130196770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 0x7fd130196770 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd11802a9c0 tx=0x7fd1180055a0 comp rx=0 tx=0).stop 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 shutdown_connections 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fd11c06aef0 0x7fd11c06d3b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1300ffd10 0x7fd130196770 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 --2- 172.21.15.82:0/649314727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1300ff360 0x7fd130196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 >> 172.21.15.82:0/649314727 conn(0x7fd130075e10 msgr2=0x7fd130067f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 shutdown_connections 2024-06-06T07:46:03.884 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:03.884+0000 7fd136ae6640 1 -- 172.21.15.82:0/649314727 wait complete. 2024-06-06T07:46:04.995 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:04 smithi184 ceph-mon[37288]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:04.995 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:04 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/649314727' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:05.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:04 smithi082 ceph-mon[29354]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:05.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:04 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/649314727' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:07.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:06 smithi082 ceph-mon[29354]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:07.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:06 smithi184 ceph-mon[37288]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:07.644 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:07.853 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:07.975 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:46:07.975 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:46:07.975 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:46:07.975 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:46:07.975 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:07 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:46:08.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T07:46:08.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T07:46:08.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T07:46:08.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' 2024-06-06T07:46:08.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:07 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:46:09.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:08 smithi082 ceph-mon[29354]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:09.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:08 smithi184 ceph-mon[37288]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:09.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 -- 172.21.15.82:0/897900277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 msgr2=0x7fb240103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:09.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 --2- 172.21.15.82:0/897900277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 0x7fb240103d60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009880 tx=0x7fb23c02a4a0 comp rx=0 tx=0).stop 2024-06-06T07:46:09.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 -- 172.21.15.82:0/897900277 shutdown_connections 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 --2- 172.21.15.82:0/897900277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 0x7fb240103d60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 --2- 172.21.15.82:0/897900277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2400ff360 0x7fb2400ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 -- 172.21.15.82:0/897900277 >> 172.21.15.82:0/897900277 conn(0x7fb240075e10 msgr2=0x7fb2400fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 -- 172.21.15.82:0/897900277 shutdown_connections 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 -- 172.21.15.82:0/897900277 wait complete. 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 Processor -- start 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.500+0000 7fb2484db640 1 -- start start 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb2484db640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2400ff360 0x7fb2401963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb2484db640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 0x7fb240196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb2484db640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb240196f90 con 0x7fb2400ffd10 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb2484db640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24019acb0 con 0x7fb2400ff360 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb245a76640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 0x7fb240196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:09.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb245a76640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 0x7fb240196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42740/0 (socket says 172.21.15.82:42740) 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb245a76640 1 -- 172.21.15.82:0/3367174156 learned_addr learned my addr 172.21.15.82:0/3367174156 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb246277640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2400ff360 0x7fb2401963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb245a76640 1 -- 172.21.15.82:0/3367174156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2400ff360 msgr2=0x7fb2401963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb245a76640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2400ff360 0x7fb2401963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb245a76640 1 -- 172.21.15.82:0/3367174156 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c009560 con 0x7fb2400ffd10 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb246277640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2400ff360 0x7fb2401963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb245a76640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 0x7fb240196900 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb23c02a9c0 tx=0x7fb23c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb2377fe640 1 -- 172.21.15.82:0/3367174156 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb23c004aa0 con 0x7fb2400ffd10 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb2377fe640 1 -- 172.21.15.82:0/3367174156 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb23c004c00 con 0x7fb2400ffd10 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24019af30 con 0x7fb2400ffd10 2024-06-06T07:46:09.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb2377fe640 1 -- 172.21.15.82:0/3367174156 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb23c003a50 con 0x7fb2400ffd10 2024-06-06T07:46:09.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.501+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24019b420 con 0x7fb2400ffd10 2024-06-06T07:46:09.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.502+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb240100840 con 0x7fb2400ffd10 2024-06-06T07:46:09.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.503+0000 7fb2377fe640 1 -- 172.21.15.82:0/3367174156 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb23c031c90 con 0x7fb2400ffd10 2024-06-06T07:46:09.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.503+0000 7fb2377fe640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb22c06aef0 0x7fb22c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:09.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.503+0000 7fb2377fe640 1 -- 172.21.15.82:0/3367174156 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb23c0aa9d0 con 0x7fb2400ffd10 2024-06-06T07:46:09.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.503+0000 7fb246277640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb22c06aef0 0x7fb22c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:09.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.504+0000 7fb246277640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb22c06aef0 0x7fb22c06d3b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb230009930 tx=0x7fb230008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:09.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.508+0000 7fb2377fe640 1 -- 172.21.15.82:0/3367174156 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb23c076030 con 0x7fb2400ffd10 2024-06-06T07:46:09.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.717+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb24004e210 con 0x7fb2400ffd10 2024-06-06T07:46:09.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.717+0000 7fb2377fe640 1 -- 172.21.15.82:0/3367174156 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb23c0400d0 con 0x7fb2400ffd10 2024-06-06T07:46:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:09.717 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.719+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb22c06aef0 msgr2=0x7fb22c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.719+0000 7fb2484db640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb22c06aef0 0x7fb22c06d3b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb230009930 tx=0x7fb230008040 comp rx=0 tx=0).stop 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.719+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 msgr2=0x7fb240196900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.719+0000 7fb2484db640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 0x7fb240196900 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb23c02a9c0 tx=0x7fb23c0055a0 comp rx=0 tx=0).stop 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.719+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 shutdown_connections 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.719+0000 7fb2484db640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb22c06aef0 0x7fb22c06d3b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.719+0000 7fb2484db640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb2400ffd10 0x7fb240196900 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.719+0000 7fb2484db640 1 --2- 172.21.15.82:0/3367174156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb2400ff360 0x7fb2401963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.719+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 >> 172.21.15.82:0/3367174156 conn(0x7fb240075e10 msgr2=0x7fb240102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.720+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 shutdown_connections 2024-06-06T07:46:09.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:09.720+0000 7fb2484db640 1 -- 172.21.15.82:0/3367174156 wait complete. 2024-06-06T07:46:10.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:09 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3367174156' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:10.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:09 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3367174156' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:11.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:10 smithi082 ceph-mon[29354]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:11.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:10 smithi184 ceph-mon[37288]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:13.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:12 smithi082 ceph-mon[29354]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:13.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:12 smithi184 ceph-mon[37288]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:13.424 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:13.648 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:15.007 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:14 smithi082 ceph-mon[29354]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:15.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:14 smithi184 ceph-mon[37288]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.221+0000 7f3203e8a640 1 -- 172.21.15.82:0/2687608572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ff3a0 msgr2=0x7f31fc0ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.221+0000 7f3203e8a640 1 --2- 172.21.15.82:0/2687608572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ff3a0 0x7f31fc0ff780 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f31f8009880 tx=0x7f31f802a4a0 comp rx=0 tx=0).stop 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.221+0000 7f3203e8a640 1 -- 172.21.15.82:0/2687608572 shutdown_connections 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.221+0000 7f3203e8a640 1 --2- 172.21.15.82:0/2687608572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31fc0ffd50 0x7f31fc103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.221+0000 7f3203e8a640 1 --2- 172.21.15.82:0/2687608572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ff3a0 0x7f31fc0ff780 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.221+0000 7f3203e8a640 1 -- 172.21.15.82:0/2687608572 >> 172.21.15.82:0/2687608572 conn(0x7f31fc0fac50 msgr2=0x7f31fc0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.221+0000 7f3203e8a640 1 -- 172.21.15.82:0/2687608572 shutdown_connections 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.221+0000 7f3203e8a640 1 -- 172.21.15.82:0/2687608572 wait complete. 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3203e8a640 1 Processor -- start 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3203e8a640 1 -- start start 2024-06-06T07:46:15.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3203e8a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31fc0ff3a0 0x7f31fc100e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3203e8a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ffd50 0x7f31fc101380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3203e8a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc1018c0 con 0x7f31fc0ffd50 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3203e8a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc101a30 con 0x7f31fc0ff3a0 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3201425640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ffd50 0x7f31fc101380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3201425640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ffd50 0x7f31fc101380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:46064/0 (socket says 172.21.15.82:46064) 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3201425640 1 -- 172.21.15.82:0/910833585 learned_addr learned my addr 172.21.15.82:0/910833585 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3201c26640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31fc0ff3a0 0x7f31fc100e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.222+0000 7f3201425640 1 -- 172.21.15.82:0/910833585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31fc0ff3a0 msgr2=0x7f31fc100e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.223+0000 7f3201425640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31fc0ff3a0 0x7f31fc100e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.223+0000 7f3201425640 1 -- 172.21.15.82:0/910833585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f8009560 con 0x7f31fc0ffd50 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.223+0000 7f3201425640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ffd50 0x7f31fc101380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00b3d0 tx=0x7f31ec00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.223+0000 7f31f2ffd640 1 -- 172.21.15.82:0/910833585 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31ec011800 con 0x7f31fc0ffd50 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.223+0000 7f31f2ffd640 1 -- 172.21.15.82:0/910833585 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31ec011e20 con 0x7f31fc0ffd50 2024-06-06T07:46:15.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.223+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31fc1a24d0 con 0x7f31fc0ffd50 2024-06-06T07:46:15.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.223+0000 7f31f2ffd640 1 -- 172.21.15.82:0/910833585 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31ec00f2e0 con 0x7f31fc0ffd50 2024-06-06T07:46:15.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.223+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31fc1a2830 con 0x7f31fc0ffd50 2024-06-06T07:46:15.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.224+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31fc100880 con 0x7f31fc0ffd50 2024-06-06T07:46:15.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.224+0000 7f31f2ffd640 1 -- 172.21.15.82:0/910833585 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f31ec010380 con 0x7f31fc0ffd50 2024-06-06T07:46:15.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.225+0000 7f31f2ffd640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f31e806aef0 0x7f31e806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:15.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.225+0000 7f3201c26640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f31e806aef0 0x7f31e806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:15.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.225+0000 7f31f2ffd640 1 -- 172.21.15.82:0/910833585 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f31ec0897e0 con 0x7f31fc0ffd50 2024-06-06T07:46:15.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.225+0000 7f3201c26640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f31e806aef0 0x7f31e806d3b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f31f8000c00 tx=0x7f31f8004120 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:15.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.229+0000 7f31f2ffd640 1 -- 172.21.15.82:0/910833585 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f31ec054e40 con 0x7f31fc0ffd50 2024-06-06T07:46:15.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.439+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f31fc101bf0 con 0x7f31fc0ffd50 2024-06-06T07:46:15.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.440+0000 7f31f2ffd640 1 -- 172.21.15.82:0/910833585 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f31ec058460 con 0x7f31fc0ffd50 2024-06-06T07:46:15.440 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:15.440 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:15.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f31e806aef0 msgr2=0x7f31e806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:15.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f31e806aef0 0x7f31e806d3b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f31f8000c00 tx=0x7f31f8004120 comp rx=0 tx=0).stop 2024-06-06T07:46:15.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ffd50 msgr2=0x7f31fc101380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:15.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ffd50 0x7f31fc101380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00b3d0 tx=0x7f31ec00b890 comp rx=0 tx=0).stop 2024-06-06T07:46:15.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 shutdown_connections 2024-06-06T07:46:15.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f31e806aef0 0x7f31e806d3b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:15.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f31fc0ffd50 0x7f31fc101380 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:15.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 --2- 172.21.15.82:0/910833585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f31fc0ff3a0 0x7f31fc100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:15.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 >> 172.21.15.82:0/910833585 conn(0x7f31fc0fac50 msgr2=0x7f31fc109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:15.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 shutdown_connections 2024-06-06T07:46:15.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:15.442+0000 7f3203e8a640 1 -- 172.21.15.82:0/910833585 wait complete. 2024-06-06T07:46:16.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:15 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/910833585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:16.146 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:15 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/910833585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:17.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:16 smithi184 ceph-mon[37288]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:17.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:16 smithi082 ceph-mon[29354]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:19.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:18 smithi184 ceph-mon[37288]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:19.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:19.199 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:18 smithi082 ceph-mon[29354]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:19.391 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:20.795 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:20 smithi082 ceph-mon[29354]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:21.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:20 smithi184 ceph-mon[37288]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:21.122 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 -- 172.21.15.82:0/2910410438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec8101ce0 msgr2=0x7f9ec810a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 --2- 172.21.15.82:0/2910410438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec8101ce0 0x7f9ec810a1d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009880 tx=0x7f9ebc02a530 comp rx=0 tx=0).stop 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 -- 172.21.15.82:0/2910410438 shutdown_connections 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 --2- 172.21.15.82:0/2910410438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec8101ce0 0x7f9ec810a1d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 --2- 172.21.15.82:0/2910410438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ec81013c0 0x7f9ec81017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 -- 172.21.15.82:0/2910410438 >> 172.21.15.82:0/2910410438 conn(0x7f9ec8075e10 msgr2=0x7f9ec80fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 -- 172.21.15.82:0/2910410438 shutdown_connections 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 -- 172.21.15.82:0/2910410438 wait complete. 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 Processor -- start 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.119+0000 7f9ed0489640 1 -- start start 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ed0489640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec81013c0 0x7f9ec80ff020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ed0489640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ec8101ce0 0x7f9ec80ff560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ed0489640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec80ffaa0 con 0x7f9ec81013c0 2024-06-06T07:46:21.123 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ed0489640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec80ffc10 con 0x7f9ec8101ce0 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ece225640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec81013c0 0x7f9ec80ff020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ece225640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec81013c0 0x7f9ec80ff020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:46088/0 (socket says 172.21.15.82:46088) 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ece225640 1 -- 172.21.15.82:0/409094987 learned_addr learned my addr 172.21.15.82:0/409094987 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ecda24640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ec8101ce0 0x7f9ec80ff560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ece225640 1 -- 172.21.15.82:0/409094987 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ec8101ce0 msgr2=0x7f9ec80ff560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ece225640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ec8101ce0 0x7f9ec80ff560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ece225640 1 -- 172.21.15.82:0/409094987 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc009560 con 0x7f9ec81013c0 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ecda24640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ec8101ce0 0x7f9ec80ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ece225640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec81013c0 0x7f9ec80ff020 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800ba50 tx=0x7f9eb800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ec77fe640 1 -- 172.21.15.82:0/409094987 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb800cb30 con 0x7f9ec81013c0 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec81a2590 con 0x7f9ec81013c0 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ec77fe640 1 -- 172.21.15.82:0/409094987 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9eb800cc90 con 0x7f9ec81013c0 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ec77fe640 1 -- 172.21.15.82:0/409094987 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb8007740 con 0x7f9ec81013c0 2024-06-06T07:46:21.124 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.120+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec81a28f0 con 0x7f9ec81013c0 2024-06-06T07:46:21.125 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.121+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec8068510 con 0x7f9ec81013c0 2024-06-06T07:46:21.125 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.122+0000 7f9ec77fe640 1 -- 172.21.15.82:0/409094987 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f9eb8014b90 con 0x7f9ec81013c0 2024-06-06T07:46:21.125 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.123+0000 7f9ec77fe640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9eb006af00 0x7f9eb006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:21.125 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.123+0000 7f9ec77fe640 1 -- 172.21.15.82:0/409094987 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9eb80895e0 con 0x7f9ec81013c0 2024-06-06T07:46:21.125 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.123+0000 7f9ecda24640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9eb006af00 0x7f9eb006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:21.125 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.123+0000 7f9ecda24640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9eb006af00 0x7f9eb006d3c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009230 tx=0x7f9ebc005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:21.127 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.128+0000 7f9ec77fe640 1 -- 172.21.15.82:0/409094987 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9eb8058700 con 0x7f9ec81013c0 2024-06-06T07:46:21.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.335+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9ec804e210 con 0x7f9ec81013c0 2024-06-06T07:46:21.335 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.335+0000 7f9ec77fe640 1 -- 172.21.15.82:0/409094987 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9eb8058290 con 0x7f9ec81013c0 2024-06-06T07:46:21.335 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:21.335 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:21.337 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.337+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9eb006af00 msgr2=0x7f9eb006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.337+0000 7f9ed0489640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9eb006af00 0x7f9eb006d3c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009230 tx=0x7f9ebc005a20 comp rx=0 tx=0).stop 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.337+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec81013c0 msgr2=0x7f9ec80ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.337+0000 7f9ed0489640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec81013c0 0x7f9ec80ff020 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800ba50 tx=0x7f9eb800bf10 comp rx=0 tx=0).stop 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.337+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 shutdown_connections 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.337+0000 7f9ed0489640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f9eb006af00 0x7f9eb006d3c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.337+0000 7f9ed0489640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ec8101ce0 0x7f9ec80ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.337+0000 7f9ed0489640 1 --2- 172.21.15.82:0/409094987 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ec81013c0 0x7f9ec80ff020 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.337+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 >> 172.21.15.82:0/409094987 conn(0x7f9ec8075e10 msgr2=0x7f9ec81026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.338+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 shutdown_connections 2024-06-06T07:46:21.338 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:21.338+0000 7f9ed0489640 1 -- 172.21.15.82:0/409094987 wait complete. 2024-06-06T07:46:21.917 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:21 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/409094987' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:22.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:21 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/409094987' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:23.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:22 smithi184 ceph-mon[37288]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:23.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:22 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:46:23.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:22 smithi082 ceph-mon[29354]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:23.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:22 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:46:24.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:25.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:24 smithi184 ceph-mon[37288]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:25.156 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:25.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:24 smithi082 ceph-mon[29354]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.840+0000 7f095b872640 1 -- 172.21.15.82:0/3114885538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ffd10 msgr2=0x7f0954103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.840+0000 7f095b872640 1 --2- 172.21.15.82:0/3114885538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ffd10 0x7f0954103d60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0944009880 tx=0x7f094402a4a0 comp rx=0 tx=0).stop 2024-06-06T07:46:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.840+0000 7f095b872640 1 -- 172.21.15.82:0/3114885538 shutdown_connections 2024-06-06T07:46:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.840+0000 7f095b872640 1 --2- 172.21.15.82:0/3114885538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ffd10 0x7f0954103d60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.840+0000 7f095b872640 1 --2- 172.21.15.82:0/3114885538 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09540ff360 0x7f09540ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.840+0000 7f095b872640 1 -- 172.21.15.82:0/3114885538 >> 172.21.15.82:0/3114885538 conn(0x7f0954075e10 msgr2=0x7f09540fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095b872640 1 -- 172.21.15.82:0/3114885538 shutdown_connections 2024-06-06T07:46:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095b872640 1 -- 172.21.15.82:0/3114885538 wait complete. 2024-06-06T07:46:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095b872640 1 Processor -- start 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095b872640 1 -- start start 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095b872640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ff360 0x7f09541952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095b872640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09540ffd10 0x7f0954195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095b872640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0954195e90 con 0x7f09540ff360 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095b872640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0954199bb0 con 0x7f09540ffd10 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095960e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ff360 0x7f09541952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095960e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ff360 0x7f09541952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:43230/0 (socket says 172.21.15.82:43230) 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.841+0000 7f095960e640 1 -- 172.21.15.82:0/589105124 learned_addr learned my addr 172.21.15.82:0/589105124 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f0958e0d640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09540ffd10 0x7f0954195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f095960e640 1 -- 172.21.15.82:0/589105124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09540ffd10 msgr2=0x7f0954195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f095960e640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09540ffd10 0x7f0954195800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f095960e640 1 -- 172.21.15.82:0/589105124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0944009560 con 0x7f09540ff360 2024-06-06T07:46:26.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f095960e640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ff360 0x7f09541952c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f095000b420 tx=0x7f095000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f094a7fc640 1 -- 172.21.15.82:0/589105124 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0950011800 con 0x7f09540ff360 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0954199e90 con 0x7f09540ff360 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f094a7fc640 1 -- 172.21.15.82:0/589105124 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0950011e20 con 0x7f09540ff360 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f094a7fc640 1 -- 172.21.15.82:0/589105124 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f095000f2e0 con 0x7f09540ff360 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.842+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095419a3b0 con 0x7f09540ff360 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.843+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0954100840 con 0x7f09540ff360 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.844+0000 7f094a7fc640 1 -- 172.21.15.82:0/589105124 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0950010380 con 0x7f09540ff360 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.844+0000 7f094a7fc640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f094006aef0 0x7f094006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.844+0000 7f094a7fc640 1 -- 172.21.15.82:0/589105124 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0950089570 con 0x7f09540ff360 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.844+0000 7f0958e0d640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f094006aef0 0x7f094006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.844+0000 7f0958e0d640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f094006aef0 0x7f094006d3b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0944009230 tx=0x7f0944005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:26.848+0000 7f094a7fc640 1 -- 172.21.15.82:0/589105124 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0950054b50 con 0x7f09540ff360 2024-06-06T07:46:27.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:26 smithi082 ceph-mon[29354]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:27.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.057+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f095404e210 con 0x7f09540ff360 2024-06-06T07:46:27.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.057+0000 7f094a7fc640 1 -- 172.21.15.82:0/589105124 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0950058170 con 0x7f09540ff360 2024-06-06T07:46:27.058 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:27.058 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:27.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.059+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f094006aef0 msgr2=0x7f094006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f094006aef0 0x7f094006d3b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0944009230 tx=0x7f0944005a20 comp rx=0 tx=0).stop 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ff360 msgr2=0x7f09541952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ff360 0x7f09541952c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f095000b420 tx=0x7f095000b8e0 comp rx=0 tx=0).stop 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 shutdown_connections 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f094006aef0 0x7f094006d3b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09540ffd10 0x7f0954195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 --2- 172.21.15.82:0/589105124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09540ff360 0x7f09541952c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 >> 172.21.15.82:0/589105124 conn(0x7f0954075e10 msgr2=0x7f0954102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 shutdown_connections 2024-06-06T07:46:27.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:27.060+0000 7f095b872640 1 -- 172.21.15.82:0/589105124 wait complete. 2024-06-06T07:46:27.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:26 smithi184 ceph-mon[37288]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:28.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:27 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/589105124' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:28.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:27 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/589105124' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:29.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:28 smithi184 ceph-mon[37288]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:29.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:28 smithi082 ceph-mon[29354]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:30.743 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:30.962 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:31.084 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:30 smithi082 ceph-mon[29354]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:31.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:30 smithi184 ceph-mon[37288]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:32.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.605+0000 7f15d7c05640 1 -- 172.21.15.82:0/1146318779 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d01004b0 msgr2=0x7f15d0100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:32.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.605+0000 7f15d7c05640 1 --2- 172.21.15.82:0/1146318779 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d01004b0 0x7f15d0100890 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009c20 tx=0x7f15c002a970 comp rx=0 tx=0).stop 2024-06-06T07:46:32.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.605+0000 7f15d7c05640 1 -- 172.21.15.82:0/1146318779 shutdown_connections 2024-06-06T07:46:32.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.605+0000 7f15d7c05640 1 --2- 172.21.15.82:0/1146318779 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15d0100e60 0x7f15d0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:32.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.605+0000 7f15d7c05640 1 --2- 172.21.15.82:0/1146318779 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d01004b0 0x7f15d0100890 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:32.618 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.605+0000 7f15d7c05640 1 -- 172.21.15.82:0/1146318779 >> 172.21.15.82:0/1146318779 conn(0x7f15d00fbd40 msgr2=0x7f15d00fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.605+0000 7f15d7c05640 1 -- 172.21.15.82:0/1146318779 shutdown_connections 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.605+0000 7f15d7c05640 1 -- 172.21.15.82:0/1146318779 wait complete. 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.606+0000 7f15d7c05640 1 Processor -- start 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.606+0000 7f15d7c05640 1 -- start start 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.606+0000 7f15d7c05640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15d01004b0 0x7f15d01963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.606+0000 7f15d7c05640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d0100e60 0x7f15d0196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.606+0000 7f15d7c05640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d0196fc0 con 0x7f15d0100e60 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.606+0000 7f15d7c05640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d019ace0 con 0x7f15d01004b0 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.606+0000 7f15d51a0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d0100e60 0x7f15d0196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.606+0000 7f15d51a0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d0100e60 0x7f15d0196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:43254/0 (socket says 172.21.15.82:43254) 2024-06-06T07:46:32.619 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.606+0000 7f15d51a0640 1 -- 172.21.15.82:0/3315852599 learned_addr learned my addr 172.21.15.82:0/3315852599 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15d59a1640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15d01004b0 0x7f15d01963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15d51a0640 1 -- 172.21.15.82:0/3315852599 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15d01004b0 msgr2=0x7f15d01963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15d51a0640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15d01004b0 0x7f15d01963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15d51a0640 1 -- 172.21.15.82:0/3315852599 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c0009900 con 0x7f15d0100e60 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15d51a0640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d0100e60 0x7f15d0196930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f15c400b3d0 tx=0x7f15c400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15ceffd640 1 -- 172.21.15.82:0/3315852599 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15c4011800 con 0x7f15d0100e60 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d019afc0 con 0x7f15d0100e60 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15ceffd640 1 -- 172.21.15.82:0/3315852599 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15c4011e20 con 0x7f15d0100e60 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15ceffd640 1 -- 172.21.15.82:0/3315852599 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15c400f2e0 con 0x7f15d0100e60 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.607+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d019b510 con 0x7f15d0100e60 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.608+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d0101990 con 0x7f15d0100e60 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.608+0000 7f15ceffd640 1 -- 172.21.15.82:0/3315852599 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f15c4010380 con 0x7f15d0100e60 2024-06-06T07:46:32.620 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.609+0000 7f15ceffd640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15b806aef0 0x7f15b806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:32.621 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.609+0000 7f15ceffd640 1 -- 172.21.15.82:0/3315852599 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f15c4089930 con 0x7f15d0100e60 2024-06-06T07:46:32.621 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.609+0000 7f15d59a1640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15b806aef0 0x7f15b806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:32.621 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.609+0000 7f15d59a1640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15b806aef0 0x7f15b806d3b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f15c00098d0 tx=0x7f15c0009800 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:32.621 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.613+0000 7f15ceffd640 1 -- 172.21.15.82:0/3315852599 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f15c4054f10 con 0x7f15d0100e60 2024-06-06T07:46:32.828 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.827+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f15d004e210 con 0x7f15d0100e60 2024-06-06T07:46:32.828 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.828+0000 7f15ceffd640 1 -- 172.21.15.82:0/3315852599 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f15c4058530 con 0x7f15d0100e60 2024-06-06T07:46:32.829 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:32.829 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15b806aef0 msgr2=0x7f15b806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15b806aef0 0x7f15b806d3b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f15c00098d0 tx=0x7f15c0009800 comp rx=0 tx=0).stop 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d0100e60 msgr2=0x7f15d0196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d0100e60 0x7f15d0196930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f15c400b3d0 tx=0x7f15c400b890 comp rx=0 tx=0).stop 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 shutdown_connections 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f15b806aef0 0x7f15b806d3b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15d0100e60 0x7f15d0196930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 --2- 172.21.15.82:0/3315852599 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15d01004b0 0x7f15d01963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 >> 172.21.15.82:0/3315852599 conn(0x7f15d00fbd40 msgr2=0x7f15d0103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 shutdown_connections 2024-06-06T07:46:32.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:32.830+0000 7f15d7c05640 1 -- 172.21.15.82:0/3315852599 wait complete. 2024-06-06T07:46:33.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:32 smithi184 ceph-mon[37288]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:33.179 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:32 smithi082 ceph-mon[29354]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:34.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:33 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3315852599' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:34.086 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:33 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3315852599' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:35.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:34 smithi184 ceph-mon[37288]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:35.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:34 smithi082 ceph-mon[29354]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:36.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:36.814 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:36.936 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:36 smithi082 ceph-mon[29354]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:37.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:36 smithi184 ceph-mon[37288]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:38.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:37 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:46:38.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:37 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:46:38.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.503+0000 7ff155d55640 1 -- 172.21.15.82:0/3420414803 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500727d0 msgr2=0x7ff150072bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:38.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.503+0000 7ff155d55640 1 --2- 172.21.15.82:0/3420414803 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500727d0 0x7ff150072bb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff140009880 tx=0x7ff14002a530 comp rx=0 tx=0).stop 2024-06-06T07:46:38.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.503+0000 7ff155d55640 1 -- 172.21.15.82:0/3420414803 shutdown_connections 2024-06-06T07:46:38.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.503+0000 7ff155d55640 1 --2- 172.21.15.82:0/3420414803 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1500730f0 0x7ff15010c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:38.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.503+0000 7ff155d55640 1 --2- 172.21.15.82:0/3420414803 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500727d0 0x7ff150072bb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:38.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.503+0000 7ff155d55640 1 -- 172.21.15.82:0/3420414803 >> 172.21.15.82:0/3420414803 conn(0x7ff1500fbd20 msgr2=0x7ff1500fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:38.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.503+0000 7ff155d55640 1 -- 172.21.15.82:0/3420414803 shutdown_connections 2024-06-06T07:46:38.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.503+0000 7ff155d55640 1 -- 172.21.15.82:0/3420414803 wait complete. 2024-06-06T07:46:38.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff155d55640 1 Processor -- start 2024-06-06T07:46:38.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff155d55640 1 -- start start 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff155d55640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1500727d0 0x7ff150198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff155d55640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500730f0 0x7ff1501989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff155d55640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff150199040 con 0x7ff1500730f0 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff155d55640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15019cd60 con 0x7ff1500727d0 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff14effd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500730f0 0x7ff1501989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff14effd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500730f0 0x7ff1501989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56056/0 (socket says 172.21.15.82:56056) 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff14effd640 1 -- 172.21.15.82:0/2564551264 learned_addr learned my addr 172.21.15.82:0/2564551264 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff14f7fe640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1500727d0 0x7ff150198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff14effd640 1 -- 172.21.15.82:0/2564551264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1500727d0 msgr2=0x7ff150198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff14effd640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1500727d0 0x7ff150198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff14effd640 1 -- 172.21.15.82:0/2564551264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff140009560 con 0x7ff1500730f0 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.504+0000 7ff14f7fe640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1500727d0 0x7ff150198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:46:38.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.505+0000 7ff14effd640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500730f0 0x7ff1501989b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff14400b420 tx=0x7ff14400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:38.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.505+0000 7ff14cff9640 1 -- 172.21.15.82:0/2564551264 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff144011800 con 0x7ff1500730f0 2024-06-06T07:46:38.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.505+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15019d040 con 0x7ff1500730f0 2024-06-06T07:46:38.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.505+0000 7ff14cff9640 1 -- 172.21.15.82:0/2564551264 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff144011e20 con 0x7ff1500730f0 2024-06-06T07:46:38.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.505+0000 7ff14cff9640 1 -- 172.21.15.82:0/2564551264 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff14400f2e0 con 0x7ff1500730f0 2024-06-06T07:46:38.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.505+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15019d590 con 0x7ff1500730f0 2024-06-06T07:46:38.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.506+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff150073f10 con 0x7ff1500730f0 2024-06-06T07:46:38.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.506+0000 7ff14cff9640 1 -- 172.21.15.82:0/2564551264 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7ff144010380 con 0x7ff1500730f0 2024-06-06T07:46:38.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.507+0000 7ff14cff9640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff13806aef0 0x7ff13806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:38.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.507+0000 7ff14cff9640 1 -- 172.21.15.82:0/2564551264 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff144089610 con 0x7ff1500730f0 2024-06-06T07:46:38.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.507+0000 7ff14f7fe640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff13806aef0 0x7ff13806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:38.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.507+0000 7ff14f7fe640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff13806aef0 0x7ff13806d3b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff140006010 tx=0x7ff140005e60 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:38.511 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.511+0000 7ff14cff9640 1 -- 172.21.15.82:0/2564551264 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff144054b50 con 0x7ff1500730f0 2024-06-06T07:46:38.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.719+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff15004e210 con 0x7ff1500730f0 2024-06-06T07:46:38.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.720+0000 7ff14cff9640 1 -- 172.21.15.82:0/2564551264 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff144058170 con 0x7ff1500730f0 2024-06-06T07:46:38.720 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:38.720 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:38.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff13806aef0 msgr2=0x7ff13806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:38.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff13806aef0 0x7ff13806d3b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff140006010 tx=0x7ff140005e60 comp rx=0 tx=0).stop 2024-06-06T07:46:38.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500730f0 msgr2=0x7ff1501989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:38.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500730f0 0x7ff1501989b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff14400b420 tx=0x7ff14400b8e0 comp rx=0 tx=0).stop 2024-06-06T07:46:38.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 shutdown_connections 2024-06-06T07:46:38.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff13806aef0 0x7ff13806d3b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:38.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1500730f0 0x7ff1501989b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:38.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 --2- 172.21.15.82:0/2564551264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1500727d0 0x7ff150198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:38.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 >> 172.21.15.82:0/2564551264 conn(0x7ff1500fbd20 msgr2=0x7ff150104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:38.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 shutdown_connections 2024-06-06T07:46:38.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:38.722+0000 7ff155d55640 1 -- 172.21.15.82:0/2564551264 wait complete. 2024-06-06T07:46:39.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:38 smithi082 ceph-mon[29354]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:39.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:38 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2564551264' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:39.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:38 smithi184 ceph-mon[37288]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:39.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:38 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2564551264' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:41.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:40 smithi184 ceph-mon[37288]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:41.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:40 smithi082 ceph-mon[29354]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:42.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:42.727 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:43.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:42 smithi184 ceph-mon[37288]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:43.138 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:42 smithi082 ceph-mon[29354]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:44.394 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 -- 172.21.15.82:0/4018304793 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7281013c0 msgr2=0x7ff7281017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:44.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 --2- 172.21.15.82:0/4018304793 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7281013c0 0x7ff7281017a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff718009c20 tx=0x7ff71802aa00 comp rx=0 tx=0).stop 2024-06-06T07:46:44.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 -- 172.21.15.82:0/4018304793 shutdown_connections 2024-06-06T07:46:44.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 --2- 172.21.15.82:0/4018304793 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff728101ce0 0x7ff72810a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:44.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 --2- 172.21.15.82:0/4018304793 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7281013c0 0x7ff7281017a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:44.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 -- 172.21.15.82:0/4018304793 >> 172.21.15.82:0/4018304793 conn(0x7ff728075e10 msgr2=0x7ff7280fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:44.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 -- 172.21.15.82:0/4018304793 shutdown_connections 2024-06-06T07:46:44.395 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 -- 172.21.15.82:0/4018304793 wait complete. 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 Processor -- start 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.375+0000 7ff72f758640 1 -- start start 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72f758640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7281013c0 0x7ff7281962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72f758640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff728101ce0 0x7ff728196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72f758640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff728196eb0 con 0x7ff728101ce0 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72f758640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72819abd0 con 0x7ff7281013c0 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72ccf3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff728101ce0 0x7ff728196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72ccf3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff728101ce0 0x7ff728196820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56074/0 (socket says 172.21.15.82:56074) 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72ccf3640 1 -- 172.21.15.82:0/3228471177 learned_addr learned my addr 172.21.15.82:0/3228471177 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72d4f4640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7281013c0 0x7ff7281962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72ccf3640 1 -- 172.21.15.82:0/3228471177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7281013c0 msgr2=0x7ff7281962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72ccf3640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7281013c0 0x7ff7281962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72ccf3640 1 -- 172.21.15.82:0/3228471177 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff718009900 con 0x7ff728101ce0 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72d4f4640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7281013c0 0x7ff7281962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72ccf3640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff728101ce0 0x7ff728196820 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00b340 tx=0x7ff71c00b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff7267fc640 1 -- 172.21.15.82:0/3228471177 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff71c011800 con 0x7ff728101ce0 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff7267fc640 1 -- 172.21.15.82:0/3228471177 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff71c011e20 con 0x7ff728101ce0 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72819aeb0 con 0x7ff728101ce0 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff7267fc640 1 -- 172.21.15.82:0/3228471177 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff71c00f2e0 con 0x7ff728101ce0 2024-06-06T07:46:44.397 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.376+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72819b400 con 0x7ff728101ce0 2024-06-06T07:46:44.398 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.377+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff728068510 con 0x7ff728101ce0 2024-06-06T07:46:44.398 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.378+0000 7ff7267fc640 1 -- 172.21.15.82:0/3228471177 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7ff71c010380 con 0x7ff728101ce0 2024-06-06T07:46:44.398 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.378+0000 7ff7267fc640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff71006aef0 0x7ff71006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:44.398 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.378+0000 7ff7267fc640 1 -- 172.21.15.82:0/3228471177 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff71c089860 con 0x7ff728101ce0 2024-06-06T07:46:44.398 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.378+0000 7ff72d4f4640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff71006aef0 0x7ff71006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:44.398 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.379+0000 7ff72d4f4640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff71006aef0 0x7ff71006d3b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff7180098d0 tx=0x7ff718009360 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:44.398 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.383+0000 7ff7267fc640 1 -- 172.21.15.82:0/3228471177 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff71c054e40 con 0x7ff728101ce0 2024-06-06T07:46:44.593 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.593+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff72804e210 con 0x7ff728101ce0 2024-06-06T07:46:44.594 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.594+0000 7ff7267fc640 1 -- 172.21.15.82:0/3228471177 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff71c058460 con 0x7ff728101ce0 2024-06-06T07:46:44.594 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:44.594 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff71006aef0 msgr2=0x7ff71006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff71006aef0 0x7ff71006d3b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff7180098d0 tx=0x7ff718009360 comp rx=0 tx=0).stop 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff728101ce0 msgr2=0x7ff728196820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff728101ce0 0x7ff728196820 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00b340 tx=0x7ff71c00b800 comp rx=0 tx=0).stop 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 shutdown_connections 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7ff71006aef0 0x7ff71006d3b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff728101ce0 0x7ff728196820 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 --2- 172.21.15.82:0/3228471177 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7281013c0 0x7ff7281962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 >> 172.21.15.82:0/3228471177 conn(0x7ff728075e10 msgr2=0x7ff7281026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 shutdown_connections 2024-06-06T07:46:44.596 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:44.596+0000 7ff72f758640 1 -- 172.21.15.82:0/3228471177 wait complete. 2024-06-06T07:46:45.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:44 smithi184 ceph-mon[37288]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:45.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:44 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3228471177' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:45.149 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:44 smithi082 ceph-mon[29354]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:45.149 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:44 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3228471177' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:46.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:45 smithi184 ceph-mon[37288]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:46.267 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:45 smithi082 ceph-mon[29354]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:48.171 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:48.378 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:48.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:48 smithi082 ceph-mon[29354]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:48.767 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:48 smithi082 ceph-mon[29354]: from='client.14646 -' entity='client.crash.smithi082' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:46:49.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:48 smithi184 ceph-mon[37288]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:49.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:48 smithi184 ceph-mon[37288]: from='client.14646 -' entity='client.crash.smithi082' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T07:46:50.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.097+0000 7f4b764b6640 1 -- 172.21.15.82:0/573061132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ff360 msgr2=0x7f4b700ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:50.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.097+0000 7f4b764b6640 1 --2- 172.21.15.82:0/573061132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ff360 0x7f4b700ff740 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009880 tx=0x7f4b6002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:46:50.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.097+0000 7f4b764b6640 1 -- 172.21.15.82:0/573061132 shutdown_connections 2024-06-06T07:46:50.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.097+0000 7f4b764b6640 1 --2- 172.21.15.82:0/573061132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b700ffd10 0x7f4b70103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:50.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.097+0000 7f4b764b6640 1 --2- 172.21.15.82:0/573061132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ff360 0x7f4b700ff740 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:50.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.097+0000 7f4b764b6640 1 -- 172.21.15.82:0/573061132 >> 172.21.15.82:0/573061132 conn(0x7f4b70075e10 msgr2=0x7f4b700fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:50.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.097+0000 7f4b764b6640 1 -- 172.21.15.82:0/573061132 shutdown_connections 2024-06-06T07:46:50.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.097+0000 7f4b764b6640 1 -- 172.21.15.82:0/573061132 wait complete. 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.097+0000 7f4b764b6640 1 Processor -- start 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b764b6640 1 -- start start 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b764b6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b700ff360 0x7f4b70196230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b764b6640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ffd10 0x7f4b70196770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b764b6640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70196d70 con 0x7f4b700ffd10 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b764b6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70196ee0 con 0x7f4b700ff360 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b67fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ffd10 0x7f4b70196770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b67fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ffd10 0x7f4b70196770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47346/0 (socket says 172.21.15.82:47346) 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b67fff640 1 -- 172.21.15.82:0/2084459139 learned_addr learned my addr 172.21.15.82:0/2084459139 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b6ffff640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b700ff360 0x7f4b70196230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b67fff640 1 -- 172.21.15.82:0/2084459139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b700ff360 msgr2=0x7f4b70196230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:50.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b67fff640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b700ff360 0x7f4b70196230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b67fff640 1 -- 172.21.15.82:0/2084459139 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b60009560 con 0x7f4b700ffd10 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b6ffff640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b700ff360 0x7f4b70196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b67fff640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ffd10 0x7f4b70196770 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800c8f0 tx=0x7f4b5800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b6dffb640 1 -- 172.21.15.82:0/2084459139 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b58004410 con 0x7f4b700ffd10 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7019ac20 con 0x7f4b700ffd10 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b6dffb640 1 -- 172.21.15.82:0/2084459139 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b58007c60 con 0x7f4b700ffd10 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b6dffb640 1 -- 172.21.15.82:0/2084459139 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b5800f450 con 0x7f4b700ffd10 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.098+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7019b170 con 0x7f4b700ffd10 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.099+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b70100840 con 0x7f4b700ffd10 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.100+0000 7f4b6dffb640 1 -- 172.21.15.82:0/2084459139 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f4b580077a0 con 0x7f4b700ffd10 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.100+0000 7f4b6dffb640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b5c06aef0 0x7f4b5c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:50.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.100+0000 7f4b6dffb640 1 -- 172.21.15.82:0/2084459139 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4b58089bc0 con 0x7f4b700ffd10 2024-06-06T07:46:50.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.100+0000 7f4b6ffff640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b5c06aef0 0x7f4b5c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:50.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.101+0000 7f4b6ffff640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b5c06aef0 0x7f4b5c06d3b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4b60000c00 tx=0x7f4b60005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:50.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.105+0000 7f4b6dffb640 1 -- 172.21.15.82:0/2084459139 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4b580551a0 con 0x7f4b700ffd10 2024-06-06T07:46:50.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.312+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4b7004e210 con 0x7f4b700ffd10 2024-06-06T07:46:50.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.312+0000 7f4b6dffb640 1 -- 172.21.15.82:0/2084459139 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4b580587c0 con 0x7f4b700ffd10 2024-06-06T07:46:50.312 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:50.312 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:50.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b5c06aef0 msgr2=0x7f4b5c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:50.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b5c06aef0 0x7f4b5c06d3b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4b60000c00 tx=0x7f4b60005a20 comp rx=0 tx=0).stop 2024-06-06T07:46:50.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ffd10 msgr2=0x7f4b70196770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:50.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ffd10 0x7f4b70196770 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800c8f0 tx=0x7f4b5800cdb0 comp rx=0 tx=0).stop 2024-06-06T07:46:50.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 shutdown_connections 2024-06-06T07:46:50.314 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f4b5c06aef0 0x7f4b5c06d3b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:50.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b700ffd10 0x7f4b70196770 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:50.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 --2- 172.21.15.82:0/2084459139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b700ff360 0x7f4b70196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:50.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 >> 172.21.15.82:0/2084459139 conn(0x7f4b70075e10 msgr2=0x7f4b700682f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:50.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 shutdown_connections 2024-06-06T07:46:50.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:50.314+0000 7f4b764b6640 1 -- 172.21.15.82:0/2084459139 wait complete. 2024-06-06T07:46:50.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:50 smithi082 ceph-mon[29354]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:50.954 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:50 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/2084459139' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:51.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:50 smithi184 ceph-mon[37288]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:51.094 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:50 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/2084459139' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:53.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:52 smithi082 ceph-mon[29354]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:53.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:52 smithi082 ceph-mon[29354]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:46:53.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:52 smithi184 ceph-mon[37288]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:53.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:52 smithi184 ceph-mon[37288]: from='mgr.14221 172.21.15.82:0/2366348715' entity='mgr.smithi082.kocfig' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T07:46:53.977 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:46:54.194 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:46:55.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:54 smithi082 ceph-mon[29354]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:55.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:54 smithi184 ceph-mon[37288]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:56.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.002+0000 7fb797278640 1 -- 172.21.15.82:0/4059277520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101d20 msgr2=0x7fb79010a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.002+0000 7fb797278640 1 --2- 172.21.15.82:0/4059277520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101d20 0x7fb79010a210 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb784009880 tx=0x7fb78402a530 comp rx=0 tx=0).stop 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.002+0000 7fb797278640 1 -- 172.21.15.82:0/4059277520 shutdown_connections 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.002+0000 7fb797278640 1 --2- 172.21.15.82:0/4059277520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101d20 0x7fb79010a210 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.002+0000 7fb797278640 1 --2- 172.21.15.82:0/4059277520 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb790101400 0x7fb7901017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.002+0000 7fb797278640 1 -- 172.21.15.82:0/4059277520 >> 172.21.15.82:0/4059277520 conn(0x7fb7900fac50 msgr2=0x7fb7900fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.002+0000 7fb797278640 1 -- 172.21.15.82:0/4059277520 shutdown_connections 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.002+0000 7fb797278640 1 -- 172.21.15.82:0/4059277520 wait complete. 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb797278640 1 Processor -- start 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb797278640 1 -- start start 2024-06-06T07:46:56.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb797278640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101400 0x7fb790196220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb797278640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb790101d20 0x7fb790196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb797278640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb790196df0 con 0x7fb790101400 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb797278640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79019ab10 con 0x7fb790101d20 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb795014640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101400 0x7fb790196220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb795014640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101400 0x7fb790196220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:32776/0 (socket says 172.21.15.82:32776) 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb795014640 1 -- 172.21.15.82:0/3865859103 learned_addr learned my addr 172.21.15.82:0/3865859103 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb794813640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb790101d20 0x7fb790196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb795014640 1 -- 172.21.15.82:0/3865859103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb790101d20 msgr2=0x7fb790196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb795014640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb790101d20 0x7fb790196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb795014640 1 -- 172.21.15.82:0/3865859103 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb784009560 con 0x7fb790101400 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb794813640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb790101d20 0x7fb790196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb795014640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101400 0x7fb790196220 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00d8d0 tx=0x7fb78c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:56.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb7827fc640 1 -- 172.21.15.82:0/3865859103 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78c009c00 con 0x7fb790101400 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb79019adf0 con 0x7fb790101400 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.003+0000 7fb7827fc640 1 -- 172.21.15.82:0/3865859103 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb78c009d60 con 0x7fb790101400 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.004+0000 7fb7827fc640 1 -- 172.21.15.82:0/3865859103 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78c0049d0 con 0x7fb790101400 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.004+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb79019b310 con 0x7fb790101400 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.005+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb790069080 con 0x7fb790101400 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.005+0000 7fb7827fc640 1 -- 172.21.15.82:0/3865859103 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb78c004c90 con 0x7fb790101400 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.005+0000 7fb7827fc640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb77c06aef0 0x7fb77c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.006+0000 7fb794813640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb77c06aef0 0x7fb77c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.006+0000 7fb7827fc640 1 -- 172.21.15.82:0/3865859103 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb78c089950 con 0x7fb790101400 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.006+0000 7fb794813640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb77c06aef0 0x7fb77c06d3b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb784009230 tx=0x7fb784005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:46:56.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.010+0000 7fb7827fc640 1 -- 172.21.15.82:0/3865859103 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb78c054f30 con 0x7fb790101400 2024-06-06T07:46:56.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.221+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb790104a20 con 0x7fb790101400 2024-06-06T07:46:56.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.221+0000 7fb7827fc640 1 -- 172.21.15.82:0/3865859103 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb78c058550 con 0x7fb790101400 2024-06-06T07:46:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:46:56.221 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:46:56.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb77c06aef0 msgr2=0x7fb77c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:56.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb77c06aef0 0x7fb77c06d3b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb784009230 tx=0x7fb784005a20 comp rx=0 tx=0).stop 2024-06-06T07:46:56.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101400 msgr2=0x7fb790196220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:46:56.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101400 0x7fb790196220 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00d8d0 tx=0x7fb78c00dd90 comp rx=0 tx=0).stop 2024-06-06T07:46:56.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 shutdown_connections 2024-06-06T07:46:56.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7fb77c06aef0 0x7fb77c06d3b0 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb784009230 tx=0x7fb784005a20 comp rx=0 tx=0).stop 2024-06-06T07:46:56.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb790101d20 0x7fb790196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:56.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 --2- 172.21.15.82:0/3865859103 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb790101400 0x7fb790196220 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:46:56.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 >> 172.21.15.82:0/3865859103 conn(0x7fb7900fac50 msgr2=0x7fb7901026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:46:56.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.223+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 shutdown_connections 2024-06-06T07:46:56.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:46:56.224+0000 7fb797278640 1 -- 172.21.15.82:0/3865859103 wait complete. 2024-06-06T07:46:56.985 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:56 smithi082 ceph-mon[29354]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:56.985 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:56 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3865859103' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:57.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:56 smithi184 ceph-mon[37288]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:57.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:56 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3865859103' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:46:59.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:46:58 smithi082 ceph-mon[29354]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:46:59.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:46:58 smithi184 ceph-mon[37288]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:47:00.010 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T07:47:00.221 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/mon.smithi082/config 2024-06-06T07:47:01.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:47:00 smithi082 ceph-mon[29354]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:47:01.093 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:47:00 smithi184 ceph-mon[37288]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:47:01.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 -- 172.21.15.82:0/2209749022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ffd10 msgr2=0x7f0f3c103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:47:01.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 --2- 172.21.15.82:0/2209749022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ffd10 0x7f0f3c103d60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009880 tx=0x7f0f3002a4a0 comp rx=0 tx=0).stop 2024-06-06T07:47:01.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 -- 172.21.15.82:0/2209749022 shutdown_connections 2024-06-06T07:47:01.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 --2- 172.21.15.82:0/2209749022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ffd10 0x7f0f3c103d60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:47:01.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 --2- 172.21.15.82:0/2209749022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f3c0ff360 0x7f0f3c0ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:47:01.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 -- 172.21.15.82:0/2209749022 >> 172.21.15.82:0/2209749022 conn(0x7f0f3c075e10 msgr2=0x7f0f3c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 -- 172.21.15.82:0/2209749022 shutdown_connections 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 -- 172.21.15.82:0/2209749022 wait complete. 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 Processor -- start 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 -- start start 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ff360 0x7f0f3c1952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.890+0000 7f0f437ae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f3c0ffd10 0x7f0f3c195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f437ae640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c195e90 con 0x7f0f3c0ff360 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f437ae640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c199bb0 con 0x7f0f3c0ffd10 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f4154a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ff360 0x7f0f3c1952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f4154a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ff360 0x7f0f3c1952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:32800/0 (socket says 172.21.15.82:32800) 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f4154a640 1 -- 172.21.15.82:0/3064053354 learned_addr learned my addr 172.21.15.82:0/3064053354 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f40d49640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f3c0ffd10 0x7f0f3c195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:47:01.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f4154a640 1 -- 172.21.15.82:0/3064053354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f3c0ffd10 msgr2=0x7f0f3c195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f4154a640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f3c0ffd10 0x7f0f3c195800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f4154a640 1 -- 172.21.15.82:0/3064053354 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f30009560 con 0x7f0f3c0ff360 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f4154a640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ff360 0x7f0f3c1952c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400ba50 tx=0x7f0f3400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f3a7fc640 1 -- 172.21.15.82:0/3064053354 <== mon.0 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3400cb30 con 0x7f0f3c0ff360 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3c199e90 con 0x7f0f3c0ff360 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f3a7fc640 1 -- 172.21.15.82:0/3064053354 <== mon.0 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f3400cc90 con 0x7f0f3c0ff360 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f3a7fc640 1 -- 172.21.15.82:0/3064053354 <== mon.0 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f34007740 con 0x7f0f3c0ff360 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.891+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3c19a3e0 con 0x7f0f3c0ff360 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.892+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3c100840 con 0x7f0f3c0ff360 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.893+0000 7f0f3a7fc640 1 -- 172.21.15.82:0/3064053354 <== mon.0 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0f34014b90 con 0x7f0f3c0ff360 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.893+0000 7f0f3a7fc640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f2406aef0 0x7f0f2406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T07:47:01.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.893+0000 7f0f40d49640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f2406aef0 0x7f0f2406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T07:47:01.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.893+0000 7f0f3a7fc640 1 -- 172.21.15.82:0/3064053354 <== mon.0 v2:172.21.15.82:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0f340895d0 con 0x7f0f3c0ff360 2024-06-06T07:47:01.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.894+0000 7f0f40d49640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f2406aef0 0x7f0f2406d3b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0f30000c00 tx=0x7f0f30004120 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T07:47:01.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:01.897+0000 7f0f3a7fc640 1 -- 172.21.15.82:0/3064053354 <== mon.0 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0f340586f0 con 0x7f0f3c0ff360 2024-06-06T07:47:02.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.109+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0f3c04e210 con 0x7f0f3c0ff360 2024-06-06T07:47:02.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.109+0000 7f0f3a7fc640 1 -- 172.21.15.82:0/3064053354 <== mon.0 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0f34058280 con 0x7f0f3c0ff360 2024-06-06T07:47:02.110 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-06-06T07:47:02.110 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T07:47:02.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.111+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f2406aef0 msgr2=0x7f0f2406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.111+0000 7f0f437ae640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f2406aef0 0x7f0f2406d3b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0f30000c00 tx=0x7f0f30004120 comp rx=0 tx=0).stop 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.111+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ff360 msgr2=0x7f0f3c1952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.111+0000 7f0f437ae640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ff360 0x7f0f3c1952c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400ba50 tx=0x7f0f3400bf10 comp rx=0 tx=0).stop 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.111+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 shutdown_connections 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.111+0000 7f0f437ae640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:6800/2676832140,v1:172.21.15.82:6801/2676832140] conn(0x7f0f2406aef0 0x7f0f2406d3b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.112+0000 7f0f437ae640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f3c0ffd10 0x7f0f3c195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.112+0000 7f0f437ae640 1 --2- 172.21.15.82:0/3064053354 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c0ff360 0x7f0f3c1952c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.112+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 >> 172.21.15.82:0/3064053354 conn(0x7f0f3c075e10 msgr2=0x7f0f3c102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.112+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 shutdown_connections 2024-06-06T07:47:02.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-06-06T07:47:02.112+0000 7f0f437ae640 1 -- 172.21.15.82:0/3064053354 wait complete. 2024-06-06T07:47:02.929 INFO:tasks.cephadm:Teardown begin 2024-06-06T07:47:02.929 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-06T07:47:02.932 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T07:47:02.958 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T07:47:02.986 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:47:02 smithi184 ceph-mon[37288]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:47:02.986 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:47:02 smithi184 ceph-mon[37288]: from='client.? 172.21.15.82:0/3064053354' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:47:02.992 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-06T07:47:02.992 DEBUG:teuthology.orchestra.run.smithi082:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-06T07:47:03.015 DEBUG:teuthology.orchestra.run.smithi184:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-06T07:47:03.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:47:02 smithi082 ceph-mon[29354]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-06-06T07:47:03.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:47:02 smithi082 ceph-mon[29354]: from='client.? 172.21.15.82:0/3064053354' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T07:47:03.050 INFO:tasks.cephadm:Stopping all daemons... 2024-06-06T07:47:03.050 INFO:tasks.cephadm.mon.smithi082:Stopping mon.smithi082... 2024-06-06T07:47:03.050 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi082 2024-06-06T07:47:03.345 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:47:03 smithi082 systemd[1]: Stopping Ceph mon.smithi082 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:47:03.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:47:03 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082[29350]: 2024-06-06T07:47:03.344+0000 7f1373544640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi082 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T07:47:03.712 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:47:03 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082[29350]: 2024-06-06T07:47:03.344+0000 7f1373544640 -1 mon.smithi082@0(leader) e2 *** Got Signal Terminated *** 2024-06-06T07:47:03.713 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:47:03 smithi082 podman[81069]: 2024-06-06 07:47:03.406265326 +0000 UTC m=+0.168187779 container died 5b5c9653fb28e96948315a57aaa2a2b3518bdf7778c0592333d5beea466142f0 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-8a28a0a, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True) 2024-06-06T07:47:04.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:47:03 smithi082 podman[81069]: 2024-06-06 07:47:03.820156473 +0000 UTC m=+0.582078917 container cleanup 5b5c9653fb28e96948315a57aaa2a2b3518bdf7778c0592333d5beea466142f0 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082, RELEASE=reef-8a28a0a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-06T07:47:04.017 INFO:journalctl@ceph.mon.smithi082.smithi082.stdout:Jun 06 07:47:03 smithi082 bash[81069]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi082 2024-06-06T07:47:04.203 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi082.service' 2024-06-06T07:47:04.261 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T07:47:04.261 INFO:tasks.cephadm.mon.smithi082:Stopped mon.smithi082 2024-06-06T07:47:04.261 INFO:tasks.cephadm.mon.smithi184:Stopping mon.smithi184... 2024-06-06T07:47:04.261 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi184 2024-06-06T07:47:04.594 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:47:04 smithi184 systemd[1]: Stopping Ceph mon.smithi184 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:47:04.862 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:47:04 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi184[37271]: 2024-06-06T07:47:04.596+0000 7f8af1259640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi184 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T07:47:04.862 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:47:04 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi184[37271]: 2024-06-06T07:47:04.596+0000 7f8af1259640 -1 mon.smithi184@1(peon) e2 *** Got Signal Terminated *** 2024-06-06T07:47:05.112 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:47:04 smithi184 podman[61220]: 2024-06-06 07:47:04.86217341 +0000 UTC m=+0.403071398 container died 551a4ac4a9ebac0b648e8affbfbd4bb9119c3b47c4732466c4626dea79710c9b (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi184, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.build-date=20240528, GIT_CLEAN=True, RELEASE=reef-8a28a0a, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-06-06T07:47:05.429 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:47:05 smithi184 podman[61220]: 2024-06-06 07:47:05.262587758 +0000 UTC m=+0.803485744 container cleanup 551a4ac4a9ebac0b648e8affbfbd4bb9119c3b47c4732466c4626dea79710c9b (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi184, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-8a28a0a, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T07:47:05.430 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Jun 06 07:47:05 smithi184 bash[61220]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-mon-smithi184 2024-06-06T07:47:05.570 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@mon.smithi184.service' 2024-06-06T07:47:05.611 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T07:47:05.611 INFO:tasks.cephadm.mon.smithi184:Stopped mon.smithi184 2024-06-06T07:47:05.611 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-06T07:47:05.612 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.0 2024-06-06T07:47:05.899 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:05 smithi082 systemd[1]: Stopping Ceph osd.0 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:47:06.267 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:05 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0[47695]: 2024-06-06T07:47:05.899+0000 7f2fc1b85640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T07:47:06.268 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:05 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0[47695]: 2024-06-06T07:47:05.899+0000 7f2fc1b85640 -1 osd.0 40 *** Got signal Terminated *** 2024-06-06T07:47:06.268 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:05 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0[47695]: 2024-06-06T07:47:05.899+0000 7f2fc1b85640 -1 osd.0 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T07:47:11.650 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:11 smithi082 podman[81287]: 2024-06-06 07:47:11.342394194 +0000 UTC m=+5.545357312 container died 67c7d5389c9a4a8e30e693e98fa23546f4223e408c5057ea4cea755da4ad2912 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0, org.label-schema.build-date=20240528, RELEASE=reef-8a28a0a, GIT_CLEAN=True, ceph=True, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-06T07:47:12.017 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:11 smithi082 podman[81287]: 2024-06-06 07:47:11.841086449 +0000 UTC m=+6.044049570 container cleanup 67c7d5389c9a4a8e30e693e98fa23546f4223e408c5057ea4cea755da4ad2912 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-06-06T07:47:12.017 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:11 smithi082 bash[81287]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0 2024-06-06T07:47:12.412 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:12 smithi082 podman[81303]: 2024-06-06 07:47:12.084730152 +0000 UTC m=+0.734884617 container remove 67c7d5389c9a4a8e30e693e98fa23546f4223e408c5057ea4cea755da4ad2912 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=reef-8a28a0a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-06T07:47:12.678 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:12 smithi082 podman[81394]: 2024-06-06 07:47:12.316301959 +0000 UTC m=+0.017359740 image pull 27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6 2024-06-06T07:47:12.679 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:12 smithi082 podman[81394]: 2024-06-06 07:47:12.443977872 +0000 UTC m=+0.145035686 container create 9047f95522da73c76ba925570ef9b90b0c09f6199fd181f7eaffa5e2e13aca4e (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-8a28a0a, org.label-schema.schema-version=1.0) 2024-06-06T07:47:12.679 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:12 smithi082 podman[81394]: 2024-06-06 07:47:12.679650497 +0000 UTC m=+0.380708268 container init 9047f95522da73c76ba925570ef9b90b0c09f6199fd181f7eaffa5e2e13aca4e (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-8a28a0a) 2024-06-06T07:47:12.935 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:12 smithi082 podman[81394]: 2024-06-06 07:47:12.6829821 +0000 UTC m=+0.384039874 container start 9047f95522da73c76ba925570ef9b90b0c09f6199fd181f7eaffa5e2e13aca4e (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-8a28a0a, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.build-date=20240528) 2024-06-06T07:47:12.935 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:12 smithi082 podman[81394]: 2024-06-06 07:47:12.788062071 +0000 UTC m=+0.489119854 container attach 9047f95522da73c76ba925570ef9b90b0c09f6199fd181f7eaffa5e2e13aca4e (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0-deactivate, RELEASE=reef-8a28a0a, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-06T07:47:13.232 INFO:journalctl@ceph.osd.0.smithi082.stdout:Jun 06 07:47:12 smithi082 podman[81394]: 2024-06-06 07:47:12.936479394 +0000 UTC m=+0.637537168 container died 9047f95522da73c76ba925570ef9b90b0c09f6199fd181f7eaffa5e2e13aca4e (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, RELEASE=reef-8a28a0a, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T07:47:13.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.0.service' 2024-06-06T07:47:13.619 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T07:47:13.619 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-06T07:47:13.620 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-06T07:47:13.620 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.1 2024-06-06T07:47:13.944 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:13 smithi082 systemd[1]: Stopping Ceph osd.1 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:47:14.267 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:13 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1[55506]: 2024-06-06T07:47:13.944+0000 7f1635202640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T07:47:14.268 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:13 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1[55506]: 2024-06-06T07:47:13.944+0000 7f1635202640 -1 osd.1 40 *** Got signal Terminated *** 2024-06-06T07:47:14.268 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:13 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1[55506]: 2024-06-06T07:47:13.944+0000 7f1635202640 -1 osd.1 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T07:47:19.710 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:19 smithi082 podman[81496]: 2024-06-06 07:47:19.446507265 +0000 UTC m=+5.598911565 container died fa484e5f3133b8c6d3f3fc467f8f6778e137c608075ab427e0f49692d5251cdd (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, RELEASE=reef-8a28a0a, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-06T07:47:20.017 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:19 smithi082 podman[81496]: 2024-06-06 07:47:19.910914073 +0000 UTC m=+6.063318411 container cleanup fa484e5f3133b8c6d3f3fc467f8f6778e137c608075ab427e0f49692d5251cdd (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-8a28a0a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-06T07:47:20.017 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:19 smithi082 bash[81496]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1 2024-06-06T07:47:20.517 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:20 smithi082 podman[81525]: 2024-06-06 07:47:20.195676168 +0000 UTC m=+0.749907548 container remove fa484e5f3133b8c6d3f3fc467f8f6778e137c608075ab427e0f49692d5251cdd (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a) 2024-06-06T07:47:20.873 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:20 smithi082 podman[81609]: 2024-06-06 07:47:20.435677907 +0000 UTC m=+0.017244568 image pull 27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6 2024-06-06T07:47:20.873 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:20 smithi082 podman[81609]: 2024-06-06 07:47:20.563246025 +0000 UTC m=+0.144812708 container create d5249adc1ff8a084581accb4f591adaf347f5f7cf1655eaf6028c9239a585842 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a, org.label-schema.build-date=20240528) 2024-06-06T07:47:20.873 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:20 smithi082 podman[81609]: 2024-06-06 07:47:20.765730951 +0000 UTC m=+0.347297630 container init d5249adc1ff8a084581accb4f591adaf347f5f7cf1655eaf6028c9239a585842 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-06-06T07:47:20.873 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:20 smithi082 podman[81609]: 2024-06-06 07:47:20.769180132 +0000 UTC m=+0.350746794 container start d5249adc1ff8a084581accb4f591adaf347f5f7cf1655eaf6028c9239a585842 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-8a28a0a) 2024-06-06T07:47:20.873 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:20 smithi082 podman[81609]: 2024-06-06 07:47:20.874058495 +0000 UTC m=+0.455625170 container attach d5249adc1ff8a084581accb4f591adaf347f5f7cf1655eaf6028c9239a585842 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , RELEASE=reef-8a28a0a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-06-06T07:47:21.267 INFO:journalctl@ceph.osd.1.smithi082.stdout:Jun 06 07:47:21 smithi082 podman[81609]: 2024-06-06 07:47:21.020753477 +0000 UTC m=+0.602320141 container died d5249adc1ff8a084581accb4f591adaf347f5f7cf1655eaf6028c9239a585842 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-8a28a0a, io.buildah.version=1.35.3, GIT_CLEAN=True) 2024-06-06T07:47:21.633 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.1.service' 2024-06-06T07:47:21.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T07:47:21.675 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-06T07:47:21.675 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-06T07:47:21.675 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.2 2024-06-06T07:47:22.005 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:21 smithi082 systemd[1]: Stopping Ceph osd.2 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:47:22.267 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:22 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2[62830]: 2024-06-06T07:47:22.004+0000 7fee2ba7b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T07:47:22.268 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:22 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2[62830]: 2024-06-06T07:47:22.004+0000 7fee2ba7b640 -1 osd.2 40 *** Got signal Terminated *** 2024-06-06T07:47:22.268 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:22 smithi082 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2[62830]: 2024-06-06T07:47:22.004+0000 7fee2ba7b640 -1 osd.2 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T07:47:27.664 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:27 smithi082 podman[81714]: 2024-06-06 07:47:27.380511116 +0000 UTC m=+5.480600425 container died 6b3a5912f0049a23aa494d24024c8ef47dec15ea367af4cd51b637a9da203f05 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-8a28a0a, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True) 2024-06-06T07:47:28.017 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:27 smithi082 podman[81714]: 2024-06-06 07:47:27.854776067 +0000 UTC m=+5.954865405 container cleanup 6b3a5912f0049a23aa494d24024c8ef47dec15ea367af4cd51b637a9da203f05 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-8a28a0a, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-06T07:47:28.017 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:27 smithi082 bash[81714]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2 2024-06-06T07:47:28.476 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:28 smithi082 podman[81731]: 2024-06-06 07:47:28.131451487 +0000 UTC m=+0.750367748 container remove 6b3a5912f0049a23aa494d24024c8ef47dec15ea367af4cd51b637a9da203f05 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2, io.buildah.version=1.35.3, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T07:47:28.767 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:28 smithi082 podman[81820]: 2024-06-06 07:47:28.379959388 +0000 UTC m=+0.017399613 image pull 27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6 2024-06-06T07:47:28.767 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:28 smithi082 podman[81820]: 2024-06-06 07:47:28.490401813 +0000 UTC m=+0.127842012 container create b28cccf10ee784f0a04b34608c321e8738b7da030a16e95a6fe024b73352f966 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-8a28a0a) 2024-06-06T07:47:28.768 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:28 smithi082 podman[81820]: 2024-06-06 07:47:28.701403476 +0000 UTC m=+0.338843689 container init b28cccf10ee784f0a04b34608c321e8738b7da030a16e95a6fe024b73352f966 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=reef-8a28a0a, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2) 2024-06-06T07:47:28.768 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:28 smithi082 podman[81820]: 2024-06-06 07:47:28.704682401 +0000 UTC m=+0.342122612 container start b28cccf10ee784f0a04b34608c321e8738b7da030a16e95a6fe024b73352f966 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=reef-8a28a0a, ceph=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-06-06T07:47:29.253 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:28 smithi082 podman[81820]: 2024-06-06 07:47:28.784371777 +0000 UTC m=+0.421811986 container attach b28cccf10ee784f0a04b34608c321e8738b7da030a16e95a6fe024b73352f966 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, RELEASE=reef-8a28a0a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-06T07:47:29.253 INFO:journalctl@ceph.osd.2.smithi082.stdout:Jun 06 07:47:28 smithi082 podman[81820]: 2024-06-06 07:47:28.957966343 +0000 UTC m=+0.595406556 container died b28cccf10ee784f0a04b34608c321e8738b7da030a16e95a6fe024b73352f966 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, RELEASE=reef-8a28a0a, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3) 2024-06-06T07:47:29.606 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.2.service' 2024-06-06T07:47:29.645 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T07:47:29.645 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-06T07:47:29.646 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-06T07:47:29.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.3 2024-06-06T07:47:29.939 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:29 smithi184 systemd[1]: Stopping Ceph osd.3 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:47:30.343 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:29 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3[43290]: 2024-06-06T07:47:29.939+0000 7fce29419640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T07:47:30.343 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:29 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3[43290]: 2024-06-06T07:47:29.939+0000 7fce29419640 -1 osd.3 40 *** Got signal Terminated *** 2024-06-06T07:47:30.343 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:29 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3[43290]: 2024-06-06T07:47:29.939+0000 7fce29419640 -1 osd.3 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T07:47:34.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:33 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:33.919+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:34.344 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:34 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:34.037+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:10.356062+0000 front 2024-06-06T07:47:10.356088+0000 (oldest deadline 2024-06-06T07:47:33.855785+0000) 2024-06-06T07:47:35.308 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:35 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:35.001+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:10.356062+0000 front 2024-06-06T07:47:10.356088+0000 (oldest deadline 2024-06-06T07:47:33.855785+0000) 2024-06-06T07:47:35.308 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:34 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:34.913+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:35.563 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:35 smithi184 podman[61340]: 2024-06-06 07:47:35.308731454 +0000 UTC m=+5.452105822 container died a4ee4825d77bf1872dc92c58937680981504f27be364655f9560e3490f438fc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-8a28a0a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-06T07:47:35.843 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:35 smithi184 podman[61340]: 2024-06-06 07:47:35.705768797 +0000 UTC m=+5.849143155 container cleanup a4ee4825d77bf1872dc92c58937680981504f27be364655f9560e3490f438fc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3, RELEASE=reef-8a28a0a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T07:47:35.844 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:35 smithi184 bash[61340]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3 2024-06-06T07:47:36.247 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:35 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:35.964+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:10.356062+0000 front 2024-06-06T07:47:10.356088+0000 (oldest deadline 2024-06-06T07:47:33.855785+0000) 2024-06-06T07:47:36.247 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:35 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:35.866+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:36.247 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:35 smithi184 podman[61369]: 2024-06-06 07:47:35.923805464 +0000 UTC m=+0.612793411 container remove a4ee4825d77bf1872dc92c58937680981504f27be364655f9560e3490f438fc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-8a28a0a, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-06T07:47:36.518 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:36 smithi184 podman[61460]: 2024-06-06 07:47:36.150598271 +0000 UTC m=+0.016767228 image pull 27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6 2024-06-06T07:47:36.518 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:36 smithi184 podman[61460]: 2024-06-06 07:47:36.249618986 +0000 UTC m=+0.115787929 container create 6381a376a19838251887b7dd5dd842316b13e18496ad50097e44e8a20b2955de (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-8a28a0a, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T07:47:36.518 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:36 smithi184 podman[61460]: 2024-06-06 07:47:36.460516425 +0000 UTC m=+0.326685381 container init 6381a376a19838251887b7dd5dd842316b13e18496ad50097e44e8a20b2955de (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, RELEASE=reef-8a28a0a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-06T07:47:36.518 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:36 smithi184 podman[61460]: 2024-06-06 07:47:36.463887437 +0000 UTC m=+0.330056381 container start 6381a376a19838251887b7dd5dd842316b13e18496ad50097e44e8a20b2955de (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=reef-8a28a0a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T07:47:36.519 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:36 smithi184 podman[61460]: 2024-06-06 07:47:36.518748194 +0000 UTC m=+0.384917152 container attach 6381a376a19838251887b7dd5dd842316b13e18496ad50097e44e8a20b2955de (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240528, GIT_CLEAN=True, io.buildah.version=1.35.3, RELEASE=reef-8a28a0a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-06-06T07:47:36.843 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:36 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:36.843+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:36.843 INFO:journalctl@ceph.osd.3.smithi184.stdout:Jun 06 07:47:36 smithi184 podman[61460]: 2024-06-06 07:47:36.716246049 +0000 UTC m=+0.582415004 container died 6381a376a19838251887b7dd5dd842316b13e18496ad50097e44e8a20b2955de (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=reef-8a28a0a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T07:47:37.093 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:36 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:36.996+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:10.356062+0000 front 2024-06-06T07:47:10.356088+0000 (oldest deadline 2024-06-06T07:47:33.855785+0000) 2024-06-06T07:47:37.298 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.3.service' 2024-06-06T07:47:37.336 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T07:47:37.336 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-06T07:47:37.336 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-06T07:47:37.337 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.4 2024-06-06T07:47:37.593 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:37 smithi184 systemd[1]: Stopping Ceph osd.4 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:47:37.986 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:37 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:37.881+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:37.987 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:37 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:37.674+0000 7f6a2dbc6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T07:47:37.987 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:37 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:37.674+0000 7f6a2dbc6640 -1 osd.4 40 *** Got signal Terminated *** 2024-06-06T07:47:37.987 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:37 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:37.674+0000 7f6a2dbc6640 -1 osd.4 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T07:47:37.987 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:37 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:37.987+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:10.356062+0000 front 2024-06-06T07:47:10.356088+0000 (oldest deadline 2024-06-06T07:47:33.855785+0000) 2024-06-06T07:47:39.343 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:39 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:39.028+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:10.356062+0000 front 2024-06-06T07:47:10.356088+0000 (oldest deadline 2024-06-06T07:47:33.855785+0000) 2024-06-06T07:47:39.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:38 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:38.910+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:40.093 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:39 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:39.955+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:40.094 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:39 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:39.995+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:10.356062+0000 front 2024-06-06T07:47:10.356088+0000 (oldest deadline 2024-06-06T07:47:33.855785+0000) 2024-06-06T07:47:40.094 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:39 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:39.995+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:16.756825+0000 front 2024-06-06T07:47:16.756810+0000 (oldest deadline 2024-06-06T07:47:39.656513+0000) 2024-06-06T07:47:41.343 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:41 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:41.026+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:10.356062+0000 front 2024-06-06T07:47:10.356088+0000 (oldest deadline 2024-06-06T07:47:33.855785+0000) 2024-06-06T07:47:41.344 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:41 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:41.026+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:16.756825+0000 front 2024-06-06T07:47:16.756810+0000 (oldest deadline 2024-06-06T07:47:39.656513+0000) 2024-06-06T07:47:41.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:40 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:40.954+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:41.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:40 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:40.954+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:42.343 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:41 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:41.977+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:42.343 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:41 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:41.977+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:42.344 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:42 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:42.041+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:10.356062+0000 front 2024-06-06T07:47:10.356088+0000 (oldest deadline 2024-06-06T07:47:33.855785+0000) 2024-06-06T07:47:42.344 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:42 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4[49693]: 2024-06-06T07:47:42.041+0000 7f6a2a9ca640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:16.756825+0000 front 2024-06-06T07:47:16.756810+0000 (oldest deadline 2024-06-06T07:47:39.656513+0000) 2024-06-06T07:47:43.344 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:43 smithi184 podman[61564]: 2024-06-06 07:47:43.191188284 +0000 UTC m=+5.630012515 container died 6f132ecc9040d6f9ef0f9700f69b533b2154ff4098b8f35e9c0cdb330a6aab50 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-8a28a0a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T07:47:43.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:42 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:42.974+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:43.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:42 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:42.974+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:43.792 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:43 smithi184 podman[61564]: 2024-06-06 07:47:43.584073032 +0000 UTC m=+6.022897266 container cleanup 6f132ecc9040d6f9ef0f9700f69b533b2154ff4098b8f35e9c0cdb330a6aab50 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-8a28a0a, org.label-schema.vendor=CentOS) 2024-06-06T07:47:43.792 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:43 smithi184 bash[61564]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4 2024-06-06T07:47:44.093 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:43 smithi184 podman[61580]: 2024-06-06 07:47:43.7932007 +0000 UTC m=+0.599283598 container remove 6f132ecc9040d6f9ef0f9700f69b533b2154ff4098b8f35e9c0cdb330a6aab50 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, RELEASE=reef-8a28a0a, ceph=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, CEPH_POINT_RELEASE=) 2024-06-06T07:47:44.094 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:43 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:43.955+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:44.094 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:43 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:43.955+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:44.371 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:44 smithi184 podman[61672]: 2024-06-06 07:47:44.102403941 +0000 UTC m=+0.085703694 container create d27464acfc6ffd4f4095a3f8b309d10d392327815aa4424ded46a502046636c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-8a28a0a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528) 2024-06-06T07:47:44.371 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:44 smithi184 podman[61672]: 2024-06-06 07:47:44.033522966 +0000 UTC m=+0.016822711 image pull 27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6 2024-06-06T07:47:44.372 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:44 smithi184 podman[61672]: 2024-06-06 07:47:44.296424717 +0000 UTC m=+0.279724453 container init d27464acfc6ffd4f4095a3f8b309d10d392327815aa4424ded46a502046636c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a, CEPH_POINT_RELEASE=) 2024-06-06T07:47:44.372 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:44 smithi184 podman[61672]: 2024-06-06 07:47:44.29970482 +0000 UTC m=+0.283004560 container start d27464acfc6ffd4f4095a3f8b309d10d392327815aa4424ded46a502046636c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-8a28a0a, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-06T07:47:44.372 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:44 smithi184 podman[61672]: 2024-06-06 07:47:44.372079355 +0000 UTC m=+0.355379115 container attach d27464acfc6ffd4f4095a3f8b309d10d392327815aa4424ded46a502046636c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a, ceph=True) 2024-06-06T07:47:44.833 INFO:journalctl@ceph.osd.4.smithi184.stdout:Jun 06 07:47:44 smithi184 podman[61672]: 2024-06-06 07:47:44.554349367 +0000 UTC m=+0.537649108 container died d27464acfc6ffd4f4095a3f8b309d10d392327815aa4424ded46a502046636c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=reef-8a28a0a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-06T07:47:45.092 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:44 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:44.914+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:45.092 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:44 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:44.914+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:45.111 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.4.service' 2024-06-06T07:47:45.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T07:47:45.148 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-06T07:47:45.148 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-06T07:47:45.148 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.5 2024-06-06T07:47:45.343 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:45 smithi184 systemd[1]: Stopping Ceph osd.5 for 355cdb60-23d7-11ef-bc9b-c7b262605968... 2024-06-06T07:47:45.843 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:45 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:45.454+0000 7f357656e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T07:47:45.843 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:45 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:45.454+0000 7f357656e640 -1 osd.5 40 *** Got signal Terminated *** 2024-06-06T07:47:45.844 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:45 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:45.454+0000 7f357656e640 -1 osd.5 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T07:47:46.343 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:45 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:45.934+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:46.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:45 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:45.934+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:47.343 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:46 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:46.906+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:47.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:46 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:46.906+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:48.343 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:47 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:47.903+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:48.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:47 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:47.903+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:48.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:47 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:47.903+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6822 osd.2 since back 2024-06-06T07:47:21.564807+0000 front 2024-06-06T07:47:21.564819+0000 (oldest deadline 2024-06-06T07:47:47.464669+0000) 2024-06-06T07:47:49.343 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:48 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:48.915+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:49.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:48 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:48.915+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:49.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:48 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:48.915+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6822 osd.2 since back 2024-06-06T07:47:21.564807+0000 front 2024-06-06T07:47:21.564819+0000 (oldest deadline 2024-06-06T07:47:47.464669+0000) 2024-06-06T07:47:50.343 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:49 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:49.910+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6806 osd.0 since back 2024-06-06T07:47:09.463097+0000 front 2024-06-06T07:47:09.463061+0000 (oldest deadline 2024-06-06T07:47:32.962891+0000) 2024-06-06T07:47:50.343 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:49 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:49.910+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6814 osd.1 since back 2024-06-06T07:47:18.863777+0000 front 2024-06-06T07:47:18.863847+0000 (oldest deadline 2024-06-06T07:47:39.963654+0000) 2024-06-06T07:47:50.344 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:49 smithi184 ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5[56291]: 2024-06-06T07:47:49.910+0000 7f356f36a640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.82:6822 osd.2 since back 2024-06-06T07:47:21.564807+0000 front 2024-06-06T07:47:21.564819+0000 (oldest deadline 2024-06-06T07:47:47.464669+0000) 2024-06-06T07:47:51.093 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:50 smithi184 podman[61773]: 2024-06-06 07:47:50.797268353 +0000 UTC m=+5.419940739 container died 6413de6deef6208689f944d75231a9ef75fb23fd9cc252e9dbd4f105db765e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, RELEASE=reef-8a28a0a, io.buildah.version=1.35.3) 2024-06-06T07:47:51.593 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:51 smithi184 podman[61773]: 2024-06-06 07:47:51.154286622 +0000 UTC m=+5.776958997 container cleanup 6413de6deef6208689f944d75231a9ef75fb23fd9cc252e9dbd4f105db765e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-8a28a0a, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux ) 2024-06-06T07:47:51.594 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:51 smithi184 bash[61773]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5 2024-06-06T07:47:51.594 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:51 smithi184 podman[61792]: 2024-06-06 07:47:51.297488808 +0000 UTC m=+0.496353846 container remove 6413de6deef6208689f944d75231a9ef75fb23fd9cc252e9dbd4f105db765e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, RELEASE=reef-8a28a0a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528) 2024-06-06T07:47:52.093 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:51 smithi184 podman[61882]: 2024-06-06 07:47:51.715251153 +0000 UTC m=+0.109104035 container create 56a5a34ed08928803ee78e17d5a6c9161bc6707913fe2a60c5e7f0317f2b37c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-8a28a0a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-06T07:47:52.094 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:51 smithi184 podman[61882]: 2024-06-06 07:47:51.624377383 +0000 UTC m=+0.018230292 image pull 27458f0d3d566c1a9cd7c1a771b7dfcff82b809f81ea1f87575ec38a76d6e2cb quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6 2024-06-06T07:47:52.094 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:51 smithi184 podman[61882]: 2024-06-06 07:47:51.884076552 +0000 UTC m=+0.277929444 container init 56a5a34ed08928803ee78e17d5a6c9161bc6707913fe2a60c5e7f0317f2b37c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a, io.buildah.version=1.35.3, org.label-schema.build-date=20240528) 2024-06-06T07:47:52.094 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:51 smithi184 podman[61882]: 2024-06-06 07:47:51.887513244 +0000 UTC m=+0.281366143 container start 56a5a34ed08928803ee78e17d5a6c9161bc6707913fe2a60c5e7f0317f2b37c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5-deactivate, RELEASE=reef-8a28a0a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T07:47:52.094 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:51 smithi184 podman[61882]: 2024-06-06 07:47:51.942382294 +0000 UTC m=+0.336235207 container attach 56a5a34ed08928803ee78e17d5a6c9161bc6707913fe2a60c5e7f0317f2b37c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, RELEASE=reef-8a28a0a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, ceph=True) 2024-06-06T07:47:52.419 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:52 smithi184 podman[61882]: 2024-06-06 07:47:52.139360373 +0000 UTC m=+0.533213265 container died 56a5a34ed08928803ee78e17d5a6c9161bc6707913fe2a60c5e7f0317f2b37c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, ceph=True, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=reef-8a28a0a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T07:47:52.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.5.service' 2024-06-06T07:47:52.689 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:52 smithi184 podman[61905]: 2024-06-06 07:47:52.653765146 +0000 UTC m=+0.508260746 container remove 56a5a34ed08928803ee78e17d5a6c9161bc6707913fe2a60c5e7f0317f2b37c3 (image=quay.ceph.io/ceph-ci/ceph@sha256:c179b5e42f03e2f879eef3c42420c05de2b55f2fef3345160a894d678e7938f6, name=ceph-355cdb60-23d7-11ef-bc9b-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, RELEASE=reef-8a28a0a, ceph=True, io.buildah.version=1.35.3) 2024-06-06T07:47:52.689 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:52 smithi184 systemd[1]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.5.service: Deactivated successfully. 2024-06-06T07:47:52.689 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:52 smithi184 systemd[1]: Stopped Ceph osd.5 for 355cdb60-23d7-11ef-bc9b-c7b262605968. 2024-06-06T07:47:52.689 INFO:journalctl@ceph.osd.5.smithi184.stdout:Jun 06 07:47:52 smithi184 systemd[1]: ceph-355cdb60-23d7-11ef-bc9b-c7b262605968@osd.5.service: Consumed 4.475s CPU time. 2024-06-06T07:47:52.703 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T07:47:52.703 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-06T07:47:52.703 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-06T07:47:52.860 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:48:03.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-06T07:48:03.298 INFO:teuthology.orchestra.run.smithi184.stdout:Deleting cluster with fsid: 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:48:09.013 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T07:48:09.047 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T07:48:09.075 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-06T07:48:09.077 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208/remote/smithi082/crash 2024-06-06T07:48:09.078 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/crash -- . 2024-06-06T07:48:09.128 INFO:teuthology.orchestra.run.smithi082.stderr:tar: /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-06T07:48:09.128 INFO:teuthology.orchestra.run.smithi082.stderr:tar: Error is not recoverable: exiting now 2024-06-06T07:48:09.130 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208/remote/smithi184/crash 2024-06-06T07:48:09.131 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/crash -- . 2024-06-06T07:48:09.170 INFO:teuthology.orchestra.run.smithi184.stderr:tar: /var/lib/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-06T07:48:09.170 INFO:teuthology.orchestra.run.smithi184.stderr:tar: Error is not recoverable: exiting now 2024-06-06T07:48:09.171 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-06T07:48:09.171 DEBUG:teuthology.orchestra.run.smithi082:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-06T07:48:09.208 INFO:teuthology.orchestra.run.smithi082.stdout:2024-06-06T07:36:47.778014+0000 mon.smithi082 (mon.0) 122 : cluster [WRN] Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-06T07:48:09.209 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-06T07:48:09.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo egrep '\[SEC\]' /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-06T07:48:09.281 DEBUG:teuthology.orchestra.run.smithi082:> sudo egrep '\[ERR\]' /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-06T07:48:09.353 INFO:teuthology.orchestra.run.smithi082.stdout:2024-06-06T07:38:06.346324+0000 mgr.smithi184.cqttux (mgr.14246) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents 2024-06-06T07:48:09.354 INFO:tasks.cephadm:Compressing logs... 2024-06-06T07:48:09.354 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-06T07:48:09.396 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-06T07:48:09.418 INFO:teuthology.orchestra.run.smithi082.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-06T07:48:09.418 INFO:teuthology.orchestra.run.smithi082.stderr:: No such file or directory 2024-06-06T07:48:09.418 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.log 2024-06-06T07:48:09.419 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-06-06T07:48:09.422 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-06T07:48:09.422 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi082.log 2024-06-06T07:48:09.422 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-06-06T07:48:09.422 INFO:teuthology.orchestra.run.smithi082.stderr: 88.8% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-06T07:48:09.423 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi082.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-06-06T07:48:09.423 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-06T07:48:09.423 INFO:teuthology.orchestra.run.smithi082.stderr: 93.6% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi082.log.gz 2024-06-06T07:48:09.423 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-06T07:48:09.423 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mgr.smithi082.kocfig.log 2024-06-06T07:48:09.423 INFO:teuthology.orchestra.run.smithi082.stderr: 84.8%/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-volume.log: -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-06T07:48:09.423 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mon.smithi082.log 2024-06-06T07:48:09.423 INFO:teuthology.orchestra.run.smithi082.stderr: 90.9% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-06T07:48:09.424 INFO:teuthology.orchestra.run.smithi184.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-06T07:48:09.424 INFO:teuthology.orchestra.run.smithi184.stderr:: No such file or directory 2024-06-06T07:48:09.424 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.log 2024-06-06T07:48:09.425 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mgr.smithi184.cqttux.log 2024-06-06T07:48:09.425 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mgr.smithi184.cqttux.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr: 89.0% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.audit.log: /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mon.smithi184.log 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mon.smithi184.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi184.log 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr: 91.0% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.audit.log.gz/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.cephadm.log: 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-06T07:48:09.428 INFO:teuthology.orchestra.run.smithi184.stderr: 83.9% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-06T07:48:09.429 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi184.log: /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-volume.log: 93.5% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi184.log.gz 2024-06-06T07:48:09.429 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mgr.smithi082.kocfig.log: /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mon.smithi082.log: 90.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-06T07:48:09.429 INFO:teuthology.orchestra.run.smithi184.stderr: 92.5% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mgr.smithi184.cqttux.log.gz 2024-06-06T07:48:09.432 INFO:teuthology.orchestra.run.smithi082.stderr: 91.0% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-06T07:48:09.433 INFO:teuthology.orchestra.run.smithi184.stderr: 90.9% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-06T07:48:09.467 INFO:teuthology.orchestra.run.smithi184.stderr: 93.0% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-06-06T07:48:09.481 INFO:teuthology.orchestra.run.smithi184.stderr: 93.0% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-06-06T07:48:09.489 INFO:teuthology.orchestra.run.smithi184.stderr: 92.5% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-06-06T07:48:09.501 INFO:teuthology.orchestra.run.smithi082.stderr: 93.1% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-06-06T07:48:09.505 INFO:teuthology.orchestra.run.smithi184.stderr: 92.9% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mon.smithi184.log.gz 2024-06-06T07:48:09.507 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-06-06T07:48:09.507 INFO:teuthology.orchestra.run.smithi184.stderr:real 0m0.096s 2024-06-06T07:48:09.507 INFO:teuthology.orchestra.run.smithi184.stderr:user 0m0.269s 2024-06-06T07:48:09.508 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m0.027s 2024-06-06T07:48:09.509 INFO:teuthology.orchestra.run.smithi082.stderr: 92.8% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-06-06T07:48:09.535 INFO:teuthology.orchestra.run.smithi082.stderr: 93.4% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-06-06T07:48:09.551 INFO:teuthology.orchestra.run.smithi082.stderr: 89.0% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mgr.smithi082.kocfig.log.gz 2024-06-06T07:48:09.694 INFO:teuthology.orchestra.run.smithi082.stderr: 90.5% -- replaced with /var/log/ceph/355cdb60-23d7-11ef-bc9b-c7b262605968/ceph-mon.smithi082.log.gz 2024-06-06T07:48:09.696 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-06-06T07:48:09.696 INFO:teuthology.orchestra.run.smithi082.stderr:real 0m0.288s 2024-06-06T07:48:09.696 INFO:teuthology.orchestra.run.smithi082.stderr:user 0m0.695s 2024-06-06T07:48:09.696 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m0.044s 2024-06-06T07:48:09.696 INFO:tasks.cephadm:Archiving logs... 2024-06-06T07:48:09.697 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208/remote/smithi082/log 2024-06-06T07:48:09.698 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-06T07:48:09.875 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208/remote/smithi184/log 2024-06-06T07:48:09.876 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-06T07:48:09.949 INFO:tasks.cephadm:Removing cluster... 2024-06-06T07:48:09.949 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 --force 2024-06-06T07:48:10.093 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:48:10.362 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 355cdb60-23d7-11ef-bc9b-c7b262605968 --force 2024-06-06T07:48:10.516 INFO:teuthology.orchestra.run.smithi184.stdout:Deleting cluster with fsid: 355cdb60-23d7-11ef-bc9b-c7b262605968 2024-06-06T07:48:10.783 INFO:tasks.cephadm:Removing cephadm ... 2024-06-06T07:48:10.783 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-06T07:48:10.800 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-06T07:48:10.815 INFO:tasks.cephadm:Teardown complete 2024-06-06T07:48:10.816 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-06T07:48:11.065 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a4dbee47777e453d9ff12c305abaaa1e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-06T07:48:11.067 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-06T07:48:11.076 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-06T07:48:11.124 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-06T07:48:11.124 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-06T07:48:11.137 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-06T07:48:11.212 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-06T07:48:11.220 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-06T07:48:11.221 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T07:48:11.223 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T07:48:11.237 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-06-06T07:48:11.238 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-06-06T07:48:11.242 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T07:48:11.242 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-06-06T07:48:11.242 INFO:teuthology.orchestra.run.smithi184.stdout:^* hv01.front.sepia.ceph.com 3 6 377 28 -381us[ -424us] +/- 39ms 2024-06-06T07:48:11.243 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -1713us[-1756us] +/- 65ms 2024-06-06T07:48:11.243 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +2876us[+2833us] +/- 79ms 2024-06-06T07:48:11.243 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T07:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-06-06T07:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv01.front.sepia.ceph.com 3 6 377 31 -387us[ -435us] +/- 39ms 2024-06-06T07:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 -1715us[-1763us] +/- 65ms 2024-06-06T07:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +2868us[+2820us] +/- 79ms 2024-06-06T07:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T07:48:11.245 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-06T07:48:11.253 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-06T07:48:11.253 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-06T07:48:11.277 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-06T07:48:11.360 DEBUG:teuthology.orchestra.run.smithi184:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-06T07:48:11.427 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T07:48:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T07:48:11.458 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T07:48:11.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T07:48:11.485 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-06T07:48:11.496 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-06T07:48:11.520 INFO:teuthology.task.internal:Duration was 1201.880004 seconds 2024-06-06T07:48:11.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-06T07:48:11.553 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-06T07:48:11.553 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-06T07:48:11.556 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-06T07:48:11.594 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T07:48:11.594 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T07:48:11.888 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-06T07:48:11.888 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-06-06T07:48:11.888 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-06T07:48:11.909 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2024-06-06T07:48:11.909 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-06T07:48:11.950 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-06T07:48:11.950 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-06T07:48:11.953 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-06T07:48:12.017 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-06T07:48:12.017 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-06T07:48:12.020 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-06T07:48:12.402 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-06T07:48:12.411 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-06T07:48:12.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-06T07:48:12.439 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-06T07:48:12.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-06T07:48:12.473 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-06T07:48:12.482 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-06T07:48:12.502 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-06-06T07:48:12.527 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2024-06-06T07:48:12.562 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-06T07:48:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T07:48:12.596 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-06T07:48:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T07:48:12.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-06T07:48:12.626 INFO:teuthology.task.internal:Transferring archived files... 2024-06-06T07:48:12.627 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208/remote/smithi082 2024-06-06T07:48:12.627 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-06T07:48:12.677 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743208/remote/smithi184 2024-06-06T07:48:12.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-06T07:48:12.712 INFO:teuthology.task.internal:Removing archive directory... 2024-06-06T07:48:12.712 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-06T07:48:12.714 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-06T07:48:12.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-06T07:48:12.772 INFO:teuthology.task.internal:Not uploading archives. 2024-06-06T07:48:12.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-06T07:48:12.796 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-06T07:48:12.796 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-06T07:48:12.799 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-06T07:48:12.811 INFO:teuthology.orchestra.run.smithi082.stdout: 266045 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 6 07:48 /home/ubuntu/cephtest 2024-06-06T07:48:12.816 INFO:teuthology.orchestra.run.smithi184.stdout: 266045 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 6 07:48 /home/ubuntu/cephtest 2024-06-06T07:48:12.817 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-06T07:48:12.827 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-06T07:48:12.899 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1201.8800039291382 failure_reason: '"2024-06-06T07:38:06.346324+0000 mgr.smithi184.cqttux (mgr.14246) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a4dbee47777e453d9ff12c305abaaa1e status: fail success: false 2024-06-06T07:48:12.899 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-06T07:48:12.984 INFO:teuthology.run:FAIL